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ABSTRACT 

The  thesis  provides  lessons  learned  from  experiments  conducted  by  the 
1 1 m  Air  Force  to  verify  the  capabilities  of  two  vendor-produced  Radar  Data 
Management  Systems  (RDMS).  The  first  part  of  the  thesis  provides  background 
information  explaining  the  impetus  for  such  experiments  and  why  a  lessons 
learned  approach  was  taken.  The  experimental  plan  and  the  final  report  from  the 
PACAF  experiments  are  analyzed  using  evaluation  tools  taught  in  the  C3 
curriculum  at  the  Naval  Postgraduate  School.  The  lessons  learned  from  the 
mistakes  made  during  these  experiments  are  applied  to  produce  a  revised 
Experimental  Plan.  A  lessons  learned  section  follows  the  analysis.  This  section 
discusses  specific  lessons  learned  from  the  1 1th  Air  Force  experiments  as  well  as 
more  general  lessons  learned  by  the  author.  The  thesis  concludes  with  two 
chapters  that  provide  overall  conclusions  and  a  summary,  and  recommendations 
for  future  work  that  can  be  accomplished  in  the  area  of  radar  data  management. 
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I.     INTRODUCTION 

Pacific  Air  Forces  (PACAF)  is  presently  acquiring  a  radar  data  management 
system  (RDMS).  This  system  is  intended  to  both  reduce  the  amount  of  radar 
clutter  presented  to  the  operator  and  reestablish  radar  sensitivity  which  was 
deliberately  reduced  for  reasons  to  be  given  below.  The  underlying  problems 
that  would  be  solved  by  acquiring  a  RDMS  were  due  to  the  fact  that  radar 
technology  was  upgraded  faster  than  the  technology  of  the  computers  that 
process  the  radar  data  in  the  air  defense  command  and  control  (C2)  architecture. 
In  1983,  a  new  generation  of  radar  became  operational.  The  gap  between  radar 
and  computer  technologies  immediately  created  a  problem;  the  collective  input  of 
radar  data  overwhelmed  the  central  computer.  To  relieve  this  burden,  engineering 
change  proposals  (ECPs)  were  implemented  to  reduce  radar  sensitivity  and,  thus, 
the  amount  of  data  forwarded  to  the  central  computer. 

The  ECPs  were  intended  as  a  temporary  solution,  but  they  are  still  being 
used.  The  ECPs  arbitrarily  filter  radar  data  and,  as  a  result,  actual  targets  may  be 
filtered  out,  and  air  defense  operations  may  be  compromised.  A  more  permanent 
solution  is  needed. 

In  1990-1991,  two  defense  contractors  claimed  to  have  developed  radar  data 
management  devices  that  exploited  new  computer  technology  to  reduced  the 
volume  of  data  in  a  more  selective  way.  By  using  these  devices,  the  radar  could 
be  restored  to  full  operational  capability,  and  the  volume  of  data  could  still  be 
kept  small  enough  for  the  existing  C2  computers  to  process.  The  possibility  of 
overloading  the  computer  would  be  small. 

PACAF  is  anxious  to  exploit  this  technological  improvement.  It  would  allow 
them  to  better  fulfill  their  mission  requirements  and  still  delay  procurement  of  a 
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new  generation  of  air  defense  computers.  Before  committing  to  this  course  of 
action,  PACAF  wanted  to  assess  the  validity  of  the  two  manufacture's  claims,  and 
identify  the  best  system. 

An  experiment  was  conducted  by  the  lltn  Air  Control  Wing  (ACW)  at 
Elmendorf  AFB,  Alaska.  The  data  collection  had  to  be  done  in  a  manner  that 
would  not  interfere  with  the  performance  of  the  Region  Operations  Control 
Center  (ROCC)  operational  mission,  but  actual  field  trials  were  preferred  to 
laboratory  tests.  As  a  result,  the  data  collection  operation  was  "piggybacked" 
onto  normal  operations. 

When  the  1 1 tn  ACW  attempted  to  analyze  this  data  to  see  whether  the 
manufacture's  claims  were  true  and  to  determine  which  manufacture's  equipment 
performed  better,  they  could  not.  They  found  that  the  massive  volume  of  data 
collected  did  not  include  the  information  necessary  to  fully  answer  the  questions: 
"Will  the  manufacture's  equipment  eliminate  data  on  tracks  necessary  for  the 
conduct  of  air  operations?  "  The  llm  ACW  needed  to  be  able  to  identify  the 
tracks,  but  that  information  was  not  available. 

The  purpose  of  this  thesis  is  to  critically  examine  this  case  and  document 
what  went  wrong.   General  questions  such  as: 

can    experiments    be    successfully    conducted    within    the    operational 
environment? 

will  be  addressed,  but  the  primary  emphasis  will  be  on  the  PACAF  experiment. 


II.  BACKGROUND 

This  chapter  provides  background  information  that  is  necessary  to 
understand  the  following  analysis  of  the  PACAF  experiment.  The  Impetus  For 
Change  Section  explains  the  radar  data  management  problems  and  why  a 
solution  is  needed.  The  System  Description  Section  describes  the  components 
directly  involved  with  radar  data  management.  A  description  of  the  RDMS  as  a 
potential  solution  along  with  the  anticipated  benefits  to  the  command  and  control 
system  are  addressed  in  the  Radar  Data  Management  Equipment  Description 
Section.  Finally,  the  Special  Considerations  Section  illustrates  the  challenges  of 
conducting  an  experiment  in  an  operational  environment. 

A.    IMPETUS  FOR  CHANGE 

The  Alaska  NORAD  Region  (ANR)  was  driven  by  several  factors  to  change 
the  way  radar  data  was  managed.  The  decline  of  the  Soviet  threat  and  resulting 
draw  down  of  the  American  military  machine  was  a  principal  factor.  In  1990,  the 
U.S.  Air  Force  began  a  slow  process  of  reorganization,  within  which  the  Alaskan 
Air  Command  (AAC)  ceased  to  be  a  major  command.  It  became  the  llm  Air 
Force,  under  the  organizational  command  of  Pacific  Air  Forces  (PACAF).  The 
reorganization  meant  that  the  1 1"1  AF  now  had  to  go  through  PACAF  for  all  of 
its  funding,  including  justification  of  the  funding  for  existing  programs. 

The  eruption  of  Mount  Pinatubo  in  Philippines  and  resulting  loss  of  Clark  Air 
Base  was  another  factor.  Clark  Air  Base  was  the  location  of  an  air  training 
exercise  called  Cope  Thunder,  a  major  exercise  PACAF  used  to  train  its  forces  in 
Asia.  PACAF  needed  to  fill  the  void  left  by  the  loss  of  Clark  AB  and  thus  Cope 
Thunder,  and  Alaska  was  seen  as  the  best  replacement.  The  Alaskan  theater 
provided  many  opportunities:   large  training  areas  over  sparsely  populated  areas; 


the  capability  to  add  more  training  areas;  in-theater  tanker  support;  two  large 
bases  capable  of  handling  an  expanding  number  of  fighter  squadrons  as  well  as 
accommodating  visiting  squadrons;  a  well  established  command  and  control 
network  (ANR  ROCC  and  AWACS  (Airborne  Warning  and  Control  System)); 
and  new  developments  for  integrating  theater  wide  command  and  control 
facilities  that  were  already  in  progress  (i.e.,  Commander's  Theater  Information 
System  (CTIS)  and  Aircraft  Combat  Maneuvering  Instrumentation  (AQVQ)). 

PACAF  decided  to  move  Cope  Thunder  to  Alaska,  and  plans  were  put 
together  to  expand  the  training  ranges.  But,  radar  coverage  within  the  proposed 
training  areas  was  not  adequate  to  provide  safe  operations.  Additional  radars 
were  needed,  and  an  initial  increment  of  two  was  requested  to  increase  low  level 
coverage.  To  be  effective,  the  additional  radars  had  to  be  integrated  into  the 
ROCC,  but  the  central  computer  is  a  weak  link  because  of  its  limited  processing 
capability.  Therefore,  adding  more  radars,  especially  radars  concerned  with  low 
level  coverage,  which  produce  more  clutter  than  radars  concerned  with  high  level 
coverage,  meant  a  much  greater  chance  of  overloading  the  ROCC  central 
computer. 

Radar  data  management  became  a  key  issue.  Given  the  factors  of  limited  time 
(PACAF  was  proceeding  rapidly  with  the  range  improvement  project),  limited 
funding  (most  of  the  funding  was  going  for  range  improvements),  limited 
computer  processing  power  of  the  ROCC  computer,  and  limited  sensitivity  of 
existing  radars  caused  by  the  ECPs;  the  ll"1  ACW  staff  knew  one  possible 
answer.  They  had  been  working  on  the  issue  of  radar  data  management  for  over 
a  year  and  knew  of  commercial-off-the  shelf  equipment  that  would  be  low  cost, 
quickly  installed,  reliable,  maintainable,  and  interoperable  with  the  existing  and 


new  radars  and  with  the  ROCC.  This  new  equipment  was  available  from  two 
defense  contractors. 

The  lltn  ACW  was  tasked  with  investigating  both  potential  solutions  and 
making  recommendations  about  which,  if  any,  system  to  use,  and  how  to  use  it. 
The  first  major  task  was  to  confirm  the  claims  made  by  the  defense  contractors. 
Experimental  plans  were  drawn  and  experiments  were  run  to  test  the  systems. 
Would  the  RDMSs  allow  the  ECPs  to  be  removed  from  the  radars  to  restore  their 
sensitivity,  and  at  the  same  time  reduce  the  amount  of  clutter  being  sent  to  the 
central  computer  without  eliminating  valid  air  tracks? 

The  1  ltn  ACW  staff  was  in  charge  of  planning  and  executing  the  experiment, 
analyzing  the  data,  and  making  recommendations  about  the  radar  data 
management  systems.  Shortly  after  the  experiments  concluded,  the  author,  who 
was  in  charge  of  the  experiment,  was  accepted  to  the  Naval  Postgraduate  School 
(NPS).  The  data  had  not  been  closely  analyzed,  and  with  the  training  received  at 
NPS,  a  thorough  analysis  finally  could  be  done.  Looking  back,  many  mistakes 
had  been  made  with  the  experiment,  so  many  that  a  comprehensive  analysis  could 
not  be  done.  Therefore,  this  thesis  will  examine  what  went  wrong,  propose  a  new 
plan,  and  provide  lessons  learned. 

B.    SYSTEM  DESCRIPTION 

The  RDMS  is  aimed  at  solving  the  radar  data  management  problem  of  the 
ANR,  specifically  the  ROCC.  The  ROCC  is  the  center  for  command  and  control 
in  Alaska.  This  command  and  control  architecture  is  described  in  the  following 
paragraphs.  First,  roles  and  missions  are  examined,  then  the  system  components 
and  their  roles  in  radar  data  management  are  addressed. 


1.  Roles  and  Missions 

Command  and  control  is  unique  in  Alaska,  literally  serving  two  masters. 
Operationally,  the  ANR  reports  directly  to  NORAD  Headquarters.  The  mission  is 
to  provide  "...warning  of  a  surprise  attack  against  North  America,  and 
surveillance,  control,  and  defense  of  our  sovereign  airspace  [Ref.  1]." 
Administratively,  the  1  ltn  Air  Force  reports  directly  to  PACAF  for  everything  it 
needs  to  carry  out  its  assigned  mission. 

2.  System  Components 

Two  components  of  the  ROCC  are  directly  involved  with  the  radar  data 
management  problems,  the  long  range  radars,  better  known  as  Minimally 
Attended  Radars  (MAR)  and  the  ROCC  central  computer  located  at  the  ROCC. 
They  are  described  below.  The  radar  data  flows  from  the  MAR  to  the  ROCC 
central  computer  via  a  commercially  leased  satellite.  The  flow  of  data  is  depicted 
in  Figure  1  [Ref.  2]. 

a.    Long  Range  Radars 

Long  range  radars  are  the  heart  of  the  Alaskan  air  defense  network. 
Figure  2  depicts  the  approximate  radar  locations  in  Alaska.  In  1983,  a  new  type 
of  radar  became  operational.  "The  AN/FPS-117  Minimally  Attended  Radar 
(MAR)  is  designed  to  provide  long-range  accurate  aircraft  identification  and 
position  data..."  The  MAR  provides  "...digital  output  messages  containing  range, 
azimuth,  and  height  information  for  radar  and  beacon  targets."  [Ref.  3]. 
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Two  radar-related  factors  contribute  to  the  radar  data  management 
problems.  The  first  factor  is  the  extreme  sensitivity  of  the  radar.  The  unclassified 
operational  characteristics  of  the  radar  provide  an  insight.  Rotating  360  degrees 
every  12  seconds,  the  radar  search  volume  is  from  5  to  200NM,  +6  to  +20 
degrees  elevation,  a  100,000  feet  altitude  limit,  range  resolution  of  .5NM,  and  a 
range  accuracy  of  .25NM  [Ref.  3]. 

The  second  contributing  factor  is  the  adverse  clutter  environment 
found  in  Alaska.     "The  clutter  environment  in  Alaska  is  difficult  for  radar 
operation  because  of  unique  terrain,  a  wide  range  of  temperatures,  extreme 
weather,  and  a  large  migratory  bird  population  [Ref.  3]. 
b.     ROCC  Central  Computer 

The  ROCC  Central  Computer  is  the  central  point  for  processing  the 
data  received  from  all  of  the  radar  sites.  This  fully  redundant  computer  is  the 
Achilles  heel  of  the  entire  data  management  process.  (Figure  3  provides  a  block 
diagram  of  the  fully  redundant  ROCC  Central  Computer  [Ref.  4].)  The  main 
problem  lies  in  the  centralized  role  the  computer  plays.  In  every  12  second 
sweep,  each  radar  forwards  all  of  the  data  detected  within  its  search  volume.  The 
data  remains  unprocessed  until  it  reaches  the  computer  at  the  ROCC.  With  18 
MARs  located  in  Alaska,  a  vast  amount  of  data  must  be  processed  by  the 
computer.  The  computer  processing  time  is  synchronized  with  the  radar  sweep 
time  of  12  seconds.  If  the  computer  is  overloaded  by  too  much  data,  then  it 
arbitrarily  "dumps"  data  in  order  to  meet  the  12  second  processing  time.  Avoiding 
overloading  the  computer  was  the  driving  force  for  the  clutter-reducing  ECPs 
originally  installed  at  the  radars. 
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A  conflicting  sequence  of  events  has  occurred:  the  new  radars 
produce  too  much  data  because  of  their  sensitivity  and  the  location  of  each 
within  a  high  clutter  environment;  the  data  from  all  18  radars  must  be  processed 
by  older,  slower  computers;  and  the  clutter  reducing  ECPs  that  were  enacted  to 
avoid  computer  overload  sacrifice  the  sensitivity  that  the  new  radars  provide. 

C.    RADAR  DATA  MANAGEMENT  EQUIPMENT  DESCRIPTION 

Two  vendors  presented  potential  solutions  to  Alaska's  radar  data 
management  problems.  Litton  Data  Systems  Advanced  Tracking  System  (ATS) 
and  Sensis  Corporations  Multi-Scan  Correlator  Model  2000  (MSC-2000).  Both 
of  these  devices  work  on  similar  principles,  based  upon  software  algorithms 
designed  to  discriminate  tracks  from  clutter. 

1.      General  Description 

Each  radar  data  management  device  is  a  computer,  with  the  hardware 
and  software  necessary  to  reduce  the  amount  of  clutter  caused  by  environmental 
factors,  while  at  same  time  maintaining  a  high  level  of  radar  detection.  The  heart 
of  each  device  is  the  tracking  algorithm.  The  specifics  of  these  algorithms  are 
considered  proprietary  to  each  vendor,  and  will  not  be  addressed  here.  Generally 
speaking,  however,  the  tracking  function  is  performed  by  an  adaptive  algorithm. 
The  algorithm  is  designed  to  detect  and  establish  a  track  within  a  certain  number 
of  radar  scans  under  varying  amounts  of  search  volume,  store  the  track 
information  in  a  file,  and  compare  new  track  information  to  the  existing  file.  The 
track  information  is  kept  in  the  file  for  a  certain  number  of  radar  misses  before  the 
track  is  dropped.  In  order  for  the  tracking  algorithm  to  be  effective,  the  radar's 
search  volume  is  divided  into  thousands  of  azimuth  and  range  cells.  The  tracking 
algorithm  uses  these  cells  to  establish  the  validity  of  a  track.    Once  a  track  is 
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established  the  RDMS  sends  only  the  track  data  back  to  the  ROCC  Central 
Computer  [Ref.  5  and  6]. 
2.      Benefits 

One  approach  to  solving  the  underlying  problem  in  Alaska's  radar  data 
management  is  decentralized  processing  of  the  radar  data.  The  Litton  Data 
Systems  and  Sensis  Corporation  devices  focus  on  this  approach,  decentralizing 
the  task  of  radar  data  processing.  If  all  18  radars  could  preprocess  their  own  raw 
data,  forwarding  only  processed  track  data,  it  would  significantly  lessen  the 
processing  burden  on  the  central  computer,  thus  lessening  the  chances  of 
overload.  Several  benefits  can  be  immediately  realized: 

a.  Restoring  sensitivity  to  the  radars  without  fear  of  overloading  the 
computer  will  provide  a  higher  level  of  operational  capability.  In  the  past,  low 
level  radar  coverage  was  limited  because  it  was  the  source  of  most  clutter.  The 
ECPs  primarily  focused  on  reducing  the  low  level  radar  beam  coverage.  But,  low 
level  coverage  would  not  be  a  problem  using  the  vendor's  devices  because  tracks 
could  be  discriminated  from  the  clutter,  and  only  track  data  would  be  passed  on 
to  the  command  and  control  computer.  Lowering  the  radar  beams  would  benefit 
both  missions  of  the  ANR:  air  defense,  including  detection  of  air-to- surface 
missiles;  and  training,  which  requires  maintaining  separation  of  aircraft  for  the 
safe  conduct  of  operations  within  the  training  range. 

b.  The  procurement  of  the  next  generation  ROCC  computer  could  be 
delayed.  The  limitations  of  the  computer  processing  would  be  overcome  not  by 
developing  a  new  ROCC  computer,  but  by  taking  advantage  of  current  computer 
technology  to  provide  decentralized  processing.  A  new  ROCC  computer  would 
take  many  years  to  develop  and  cost  more  money  than  anyone  would  want  to 
spend   for  an   air  defense  command   and  control   system.      Research   and 
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development  for  the  vendor's  RDMS  is  already  accomplished— the  systems 
already  exist.  The  only  task  would  be  to  ascertain  the  validity  of  the  claims  being 
made  about  these  systems  and  determine  which  system  best  solves  the  radar  data 
management  problems. 

c.  Integration  and  interpretability  are  additional  benefits.  By  reducing 
the  workload  of  the  central  computer,  additional  radars  could  be  easily  integrated 
into  the  existing  air  defense  network.  This  is  especially  important  because  the 
range  improvement  plans  call  for  adding  two  additional  radars.  The  radars  under 
consideration  are  Westinghouse  radars  developed  for  the  Marines  and  used  to 
provide  short  range,  low  level  radar  coverage.  Both  vendor's  RDMS  are 
interoperable  with  a  wide  range  of  radars.  This  will  provide  flexibility  when 
acquiring  additional  radars  for  future  expansion  of  radar  coverage.  Deployed 
radars  equipped  with  an  RDMS  device  would  also  be  able  to  provide  track  data 
to  the  ROCC  computer  for  display. 

D.       SPECIAL  CONSIDERATIONS 

Operational  mission  constraints  and  legal  ramifications  are  necessary 
background.  Special  considerations,  driven  by  the  operational  mission,  had  to  be 
accounted  for  in  the  planning  and  execution  of  the  experiment.  Whether  or  not 
these  factors  had  a  direct  effect  on  the  outcome  of  the  experiment  is  hard  to  say. 
However  these  factors  are  worthy  of  mention.  They  illustrate  the  challenges  of 
conducting  experiments  in  the  operational  environment.  Legal  ramifications  were 
always  a  topic  of  utmost  importance.  The  commander  emphasized  that  he  did  not 
want  to  give  the  perception  that  the  lltn  AF  was  in  the  business  of  conducting 
its  own  test  and  evaluation.  Additionally,  he  did  not  want  to  give  the  impression 
that  he  was  bypassing  the  acquisition  process.     Therefore,  he  informed  all 
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involved  with  this  project  to  make  it  clear:     1  ltn  AF  had  not  gone  into  business 
for  itself. 

1.      Operational  mission  constraints 

The  experiments  would  have  to  be  conducted  on  a  non-interference 
basis.  The  operational  missions  could  not  be  sacrificed  at  the  expense  of  the 
experiments.  This  criterion  was  the  ultimate  requirement  and  provided  the 
boundaries  for  the  experiment.  A  direct  cause  and  effect  relationship  can  be  seen 
between  the  operational  mission  constraints  and  variable  selection  in  the 
experiment.  Furthermore,  controlling  factors  in  an  operational  environment  is 
much  more  difficult  than  in  a  laboratory. 

The  1 1 tn  ACW  staff  wanted  to  vary  certain  factors,  but  was  constrained 
by  operational  requirements.  The  factors  that  were  affected  were  site  selection, 
time,  and  type  of  radar.  Only  three  sites  were  allowed  as  part  of  the  experiment. 
Even  these  sites  were  subject  to  being  removed  if  they  were  needed  to  fulfill  an 
operational  mission.  The  effect  this  had  on  the  experiment  was  that  it  was  now 
limited  to  the  terrain,  air  traffic  volume,  and  clutter  unique  to  the  three  areas 
selected.  This  lead  to  several  questions:  Is  this  sample  representative  of  the 
Alaskan  environment  as  a  whole?  Could  valid  conclusions  be  drawn  for  the 
entire  area  of  responsibility  or  just  those  sections?  What  about  the  areas  where 
the  new  radars  would  be  used?  Since  none  of  the  radars  selected  is  located  in  the 
proposed  range  improvement  area,  would  the  conclusions  drawn  from  where  the 
experiments  were  conducted  be  valid  there? 

Time  was  also  affected  by  operational  constraints.  The  time  of  year  was 
limited  to  the  early  winter  season.  Would  this  be  a  representative  sample  for  the 
type  of  traffic  volume  and  clutter  seen  throughout  the  entire  year?  Additionally, 
the  time  of  day  was  limited.   The  experiment  could  only  be  conducted  before  or 
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after  the  daily  training  sorties  were  flown.  This  meant  either  early  morning  or  late 
afternoon/early  evening.  Again,  would  this  represent  an  environment  similar  to 
one  normally  seen? 

The  radars  used  were  a  factor  that  should  have  been  varied  but  could 
not  be.  Ideally,  two  different  radars  should  have  been  examined:  the  MAR  radar 
and  the  Marine  short  range  radar.  However,  since  the  Marine  radar  had  not  been 
procured,  only  the  MAR  radar  could  be  examined. 

Weather,  traffic  volume,  terrain,  and  clutter  environment  are  all  factors 
that  the  experimenter  would  like  to  either  control  or  vary  but  were  literally  at  the 
mercy  of  the  operational  constraints.  These  factors  are  all  functions  of  the  site 
selection  and  time.  Therefore,  holding  them  constant  or  allowing  them  to  vary 
was  impossible  and  reflected  the  difficulty  of  conducting  controlled  experiments 
in  an  operational  environment. 
2.      Legal  Ramifications 

The  term  "demonstration  plan"  was  used  to  emphasize  the  point  that 
operational  units  were  not  in  the  business  of  test  and  evaluation.  The  1  ltn  AF 
was  conducting  a  demonstration  of  the  equipment,  at  no  cost  or  legal  obligation 
to  the  government.  Furthermore,  the  staff  was  going  to  examine  the  data  the 
vendors  collected  to  verify  the  claims  of  the  vendors.  The  llm  AF  did  not  want 
to  give  the  perception  that  this  unit  was  bypassing,  or  undermining,  established 
acquisition  procedures.  In  fact,  the  unit  did  not  actively  pursue  vendors.  The 
vendors  made  their  own  solicitation  and  offers  of  demonstration,  at  no  cost  to  the 
government,  and  these  offers  were  made  prior  to  the  range  improvement  project. 

Up  until  this  point,  Litton  Data  Systems  was  the  only  vendor  known  to 
have  a  radar  data  management  device.  Not  until  after  completion  of  the  Litton 
Demonstration  was  it  learned  that  there  was  another  vendor  that  had  a  similar 
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device  and  wanted  the  opportunity  to  demonstrate  it  in  Alaska.  This  created  a 
potential  legal  problem.  If,  Sensis  Corporation,  the  other  vendor,  wanted  to 
demonstrate  their  equipment  and  were  turned  down,  then  the  government  could 
face  a  lawsuit.  In  order  to  afford  Sensis  the  same  opportunity  as  Litton,  it  was 
decide  that  Sensis  could  demonstrate  their  equipment.  Because  of  the  short 
notice  of  the  Sensis  demonstration,  another  Demonstration  Plan  was  not 
produced.  Instead,  the  ATS  Plan  was  followed  and  the  results  of  both 
demonstrations  were  published  in  the  Long  Range  Radar  (LRR)  Enhancements 
Staff  Study. 

E.   CONCLUSIONS 

This  chapter  provided  background  information  and  serves  as  a  prelude  to  the 
analysis  of  the  experimental  plan  and  report.  The  Impetus  for  Change  Section 
illustrated  why  the  1  lm  AF  felt  management  of  the  radar  data  was  important.  The 
System  Description  Section  detailed  the  components  involved  with  radar  data 
management.  The  shortcomings  as  well  as  the  potential  benefits  of  installing  an 
RDMS  were  elaborated  in  the  Radar  Data  Management  Equipment  Section. 
Finally,  the  chapter  ended  with  a  discussion  of  the  special  considerations  and 
their  affect  on  the  experiment.  With  the  background  firmly  established,  an 
analysis  of  the  experiment  can  be  conducted.  The  experimental  plan  and  report 
will  be  analyzed  in  Chapter  III. 
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III.     ANALYSIS 

This  chapter  presents  an  analysis  of  the  Advanced  Tracking  System  (ATS) 
Demonstration  Plan  and  the  LRR  Enhancements  Staff  Study  [Ref.  7  and  8].  From 
this  point  on,  the  ATS  Demonstration  Plan  will  be  referred  to  as  the  Plan  and  the 
LRR  Enhancements  Staff  Study  will  be  referred  to  as  the  Report.  The  reader  is 
encouraged  to  read  both  the  Plan  and  the  Report  which  can  be  found  in 
Appendices  B  and  C.  This  will  aid  in  understanding  the  analysis. 

The  Plan  and  Report  analyses  follow  a  simple  format-mistakes  are  pointed 
out  followed  by  potential  solutions.  An  Experiment  Plan  and  Reports  Format  and 
an  Evaluation  of  Studies  Paradigm  that  were  presented  in  the  C3  curriculum  at 
the  Naval  Postgraduate  School  were  used  to  analyze  both  the  Plan  and  the 
Report  [Ref.  9].  These  tools  are  used  to  identify  mistakes  and  to  provide 
guidance  for  the  solutions.  The  Experiment  Plan  and  Reports  Format  and  the 
Evaluation  of  Studies  Paradigm  are  included  as  Appendices  D  and  E.  At  the 
conclusion  of  the  analysis,  a  newly  revised  Plan  is  presented  that  integrates  all  of 
the  lessons  learned  in  the  Plan  analysis  (Appendix  F).  However,  there  is  no  newly 
revised  Report.  The  main  reason  is  that  the  data  necessary  to  provide  a  thorough 
analysis  is  not  available;  therefore,  major  sections  of  the  report  would  be  missing. 

The  Experiment  Plan  and  Reports  Format  serves  as  a  model  of  how  a  plan 
and  report  could  be  organized  and  what  elements  should  be  contained  in  the 
structures  of  plans  and  reports.  From  this  point  on,  the  Experiment  Plan  and 
Reports  Format  will  be  known  as  the  model  (Appendix  D). 

The  Evaluation  of  Studies  Paradigm  (Paradigm)  is  an  essential  ingredient  in 
the  analysis  because  it  defines  the  elements  of  the  model  (Appendix  E).  Together 
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the  model  and  the  paradigm  provide  the  framework  needed  to  analyze  the  Plan 
and  Report. 

A.    PLAN  ANALYSIS 

The  ATS  Demonstration  Plan  lacks  a  clear  statement  of  the  purpose  of  the 
experiment.  Neither  the  real  world  problems  that  the  ATS  is  intended  to  solve, 
nor  how  the  results  of  the  demonstration  will  be  used  to  show  that  it  can,  are 
clearly  stated.  As  a  result,  the  following  sections  lack  foundation.  They  should 
give  details  about  the  conduct  of  the  experiment  and  the  analysis  of  the  results, 
and  show  how  everything  is  specifically  organized  to  answer  the  questions  at 
hand.  Without  clear  articulation  of  the  problem,  this  is  not  possible.  Detailed 
analysis  of  each  section,  as  defined  in  the  model,  followed  by  a  sample 
introductory  revision  is  given  below. 

1.     Mistakes  (Introduction) 

An  experimental  plan  should  be  a  cohesive  document  that  lays  the 
foundation  for  the  entire  experiment,  up  to  and  including  the  analysis.  It  should 
provide  introductory  material  saying  who  is  performing  the  experiment,  why  the 
experiment  is  being  performed  (purpose),  and  defining  the  experimental  scope. 
This  material  should  clearly  identify  the  real  world  problem  that  the  experiment 
will  help  solve  and  the  specific  questions  the  experiment  seeks  to  answer. 

The  major  problem  with  the  Plan  is  its  lack  of  a  unified  introduction 
section  containing  clearly  defined  objectives  of  what  the  experiment  is  supposed 
to  do.  The  introductory  elements  that  the  Plan  does  contain  are  scattered 
throughout  the  plan  as  opposed  to  being  incorporated  into  a  unified  introduction 
section.  For  example:  the  team  is  introduced  in  section  2-1;  the  purpose 
statement  is  found  in  section  1  - 1 ;  and  the  objectives  are  found  in  section  4- 1 .  This 
creates  a  lack  of  focus  for  the  entire  experiment. 
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The  purpose  is  not  fully  addressed.  There  is  a  purpose  statement,  but  it 
is  rather  weak.  The  Plan  states  the". ..demonstration  is  to  determine  whether  or 
not  the  ATS  can  be  used  in  the  Alaskan  Theater  to  improve  radar  performance 
and  make  recommendations  concerning  the  desirability  of  [incorporating]  the 
ATS  in  the  Alaskan  Surveillance  and  Command  and  Control  System."  The 
purpose  also  mentions  that  the  primary  objective  is  to  "...collect  sufficient  data  to 
determine  the  ATS's  capability  for  substantially  improving  the  probability  of 
detecting  targets  (at  lower  altitudes  and  clutter  environment)."  But,  the  real 
world  problem  is  neither  poor  radar  performance  nor  low  probability  of  detection. 
The  actual  real  world  problem  is  radar  data  overloading  the  central  computer. 
This  is  not  mentioned.  Furthermore,  there  are  no  specific  questions  that  the 
experiment  seeks  to  answer.  So  naturally,  the  approach  the  experiment  will  use 
to  answer  the  research  questions  is  not  addressed  and  neither  are  the  anticipated 
results.  The  scope  of  the  experiment  is  also  not  addressed.  This  section  should 
explain  why  this  experiment  is  important  to  the  success  of  the  mission,  what 
constraints  are  imposed  on  the  experiment,  and  what  the  impact  of  the 
experiment  will  be. 

The  stated  objectives  are  not  appropriate  because  they  are  benefits 
rather  than  objectives,  and  they  do  not  map  back  to  the  primary  objective  listed  in 
the  purpose  statement:  to  provide  evidence  as  to  whether  or  not  the  ATS  is 
capable  of  improving  the  probability  of  detecting  low  level  targets  in  a  high 
clutter  environment. 

Objective  one  states: 

Reduce  demands  on  the  FYQ-93  ROCC  central  computer:  processing  time 
and  memory  space.  At  present,  central  computer  processing  time  and  memory 
storage  space  is  nearly  saturated;  while  memory  storage  in  the  processor 
controller  is  overloaded  to  the  point  it  "dumps"  data. 


19 


Objective  three  states: 

Increase  probability  of  detection  by  reducing  human  variability  in  the 
detection/tracking  process.  The  ATS  serving  as  a  front  end  processor  at  the 
radar  will  automatically  detect,  establish,  and  pass  target  data  to  the  Region 
Operations  Control  Center  (ROCC).  The  presentation  of  only  true  targets  in 
the  ROCC  will  eliminate  operator  decisions  concerning  the  validity  of  targets. 
This  will  increase  the  probability  of  establishing  air  tracks.  Also  because  only 
true  targets  are  presented,  maintaining  track  continuity  will  improve  for  both 
computer  tracking  algorithms  and  when  operators  override  computer  tracking. 

Neither  one  of  these  are  valid  objectives.  Both  of  them  are  benefits  that 
would  result  if  the  ATS  works.  However,  determining  the  truth  of  this  assumption 
is  the  primary  objective  of  the  experiment. 

Objective  two  states: 

Radar  Enhancement:  Regain  loss  of  probability  of  detection,  sacrificed  at 
the  expense  of  clutter  reduction  solutions  to  the  radar.  With  the  ATS's  ability 
to  extract  true  targets  from  clutter,  the  radar  beam  could  be  lowered  thus 
increasing  the  probability  of  detection  of  low  level  targets. 

This  objective  is  not  appropriate  because  it  is  the  approach  that  the 
experimenters  are  taking  to  test  the  ATS's  capabilities.  In  order  to  validate  the 
manufacture's  claims  about  the  ATS,  the  radar  must  be  allowed  to  operate  in  its 
fully  operational  state  uninhibited  by  any  ECPs.  The  purpose  is  determine  the 
amount  of  data  being  processed  by  the  radar  with  and  without  the  ATS,  and  to 
determine  whether  or  not  the  ATS  is  eliminating  valid  target  data. 

Objective  four  states: 

Role  of  the  ATS  in  Alaska:  By  analyzing  the  data  collected  during  the 
demonstration,  we  will  draw  inferences  concerning  the  utility  of  the  ATS  in 
Alaskan  Theater.  Areas  of  interest:  advantages  of  incorporating  the  ATS  at 
each  AN/FPS-1 17  in  Alaska  from  a  sensor  interoperability  standpoint;  added 
measure  of  safety  in  large  scale  exercises. 

This  objective  is  neither  a  benefit  nor  an  approach.    Instead  objective 

four  is  a  conclusion  that  can  only  be  made  after  the  data  is  collected  and  the 

results  analyzed. 
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2.      Solutions  (Introduction) 

The  major  problem  with  the  Plan  is  its  lack  of  a  unified  introduction 
section  containing  clearly  defined  objectives  of  what  the  Plan  is  supposed  to  do. 
The  solution  is  to  create  an  introduction  section  that  corrects  the  mistakes.  An 
example  of  what  the  introduction  section  should  look  like  follows: 

I.  REVISED  INTRODUCTION 

A.    Project  Management 

The  1 1  ACW  is  going  to  conduct  an  experiment  to  assess  the 
capabilities  of  the  Litton  Data  Systems  Advanced  Tracking  System 

(ATS). 

1.  In  order  to  manage  the  demonstration,  an  Advanced 
Tracking  System  Project  Management  Team  was  established.  The 
demonstration  team  leader  is  1  ltn  ACW/DOX;  other  team  members 
are  listed  in  paragraph  three. 

2.  The  following  agencies  are  participating  or  providing 
support  for  the  ATS  demon stration: 

a.  11  ACW/DOXXQ  --  Capt  Pierce 

b.  11  ACW/DOP  --  SSgtMyher 

c.  1 1  ACW/LGOR  --  MSgt  DeLuca 

d.  1 1  ACW/LGKC  --  MSgt  Shuler 

e.  11  ACW/LGKM  --  TSgtCushman 

f.  744ADS/DOO  --  Lt  McNeil 

g.  21stTFW/DOW       --  Capt  Hill 
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B.  Purpose 

The  purpose  of  this  experiment  will  be  to  determine  whether 
or  not  Litton  Data  Systems  Advanced  Tracking  System  (ATS)  is 
capable  of  more  efficiently  managing  the  flow  of  radar  data  coming 
into  the  Region  Operations  Control  Center  (ROCC).  Managing  the 
radar  data  more  efficiently  will  enable  the  operators  to  realize  the 
full  operational  capabilities  of  the  new  radars  without  overloading 
the  ROCC  Central  Computer.  As  a  result,  the  demonstration  is 
designed  to  provide  insight  into  the  radar  data  management 
problems  being  experienced  by  the  Alaska  NORAD  Region  (ANR) 
and  the  ability  of  the  ATS  to  solve  them.  Specifically,  the  ANR 
would  like  to  determine  the  validity  of  the  claims  of  the  ATS's 
capabilities.  For  example:  Can  the  ATS  accurately  discriminate  true 
targets  from  clutter?  From  an  air  defense  point  of  view,  can  the  ATS 
provide  target  discrimination  without  target  elimination? 

The  demonstration  will  select  three  sites,  all  equipped  with 
ECPs.  All  of  the  ECPs  will  be  turned  off  for  the  demonstration  in 
order  to  reestablish  the  radars  to  a  fully  sensitized  state.  The  ATS 
must  prove  that  it  is  capable  of  handling  the  data  workload  from  a 
fully  sensitized  radar. 

The  ATS  will  not  be  installed  at  the  selected  radars  because 
of  the  logistics  and  prohibitive  costs.  Instead,  the  ATS  equipment 
will  be  installed  in  the  ROCC. 

The  ATS  will  be  connected  at  the  ROCC  Communications 
Segment,  the  point  where  the  data  arrives  in  the  ROCC  via  satellite 
from  the  selected  radars.  The  data  arriving  at  this  point  has  not  yet 
been  processed  by  the  ROCC  central  computer.  Therefore, 
connecting  the  ATS  at  the  ROCC  Communications  Segment 
simulates  the  ATS  processing  the  data  at  the  radar  site. 

The  next  step  will  be  to  compare  the  amount  of  data  a  fully 
sensitized  radar  forwards  with  the  amount  of  data  a  fully  sensitized 
radar  equipped  with  an  ATS  forwards.    The  ROCC  provides  the 
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capability  to  simultaneously  measure  ATS  and  non-ATS  data  from 
the  same  site  (see  Figure  4)[Ref.  10].  The  reduction  in  data,  and  the 
quality  of  the  data  must  be  analyzed  simultaneously.  The  ATS  may 
live  up  to  its  clutter  reduction  claims,  but  the  data  has  to  be 
analyzed  to  ensure  that  the  ATS  is  not  eliminating  or  erroneously 
manipulating  valid  air  targets  necessary  for  air  defense  and  air 
training  missions.  In  order  to  verify  the  target  data,  a  specific 
number  of  aircraft,  flying  different  profiles,  will  be  used.  The 
experiment  will  be  designed  to  control  the  number  of  aircraft. 
However,  the  actual  flight  profiles  along  with  the  number  of  aircraft 
used  during  the  test  will  only  be  known  by  a  third  party, 
disassociated  from  the  experiment.  Only  after  the  experiment  is 
completed  and  the  analysis  turned  in  will  the  aircraft  portion  of  the 
experiment  be  revealed.  The  purpose  of  this  single  blind  design  is 
to  eliminate  any  bias  from  the  experimenters. 

The  anticipated  results:  Measurable  differences  between  the 
quantity  of  data  received  from  a  radar  in  a  fully  sensitized  state, 
when  equipped  with  an  ATS  and  when  not,  are  expected. 
However,  whether  or  not  the  ATS  eliminates  valid  air  targets 
remains  to  be  seen. 
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C.  The  scope  of  the  demonstration: 

This  demonstration  will  provide  the  information  the  1 1th  AF 
Commander  needs  to  decide  whether  or  not  the  ATS  (or  other 
associated  Radar  Data  Management  System)  provides  viable 
solutions  to  the  radar  data  management  problems  experienced  by 
the  ANR.  There  are  several  potential  benefits  if  the  ATS 
demonstration  proves  successful.  First,  ECPs  can  be  eliminated  from 
the  radars,  restoring  them  to  their  full  operational  capability,  and 
providing  greater  low  level  detection  of  aircraft  and  missiles.  In  the 
area  of  training,  the  improved  detection  capability  will  increase  the 
radar  coverage  in  the  low  level  training  areas. 

Benefits  should  also  be  seen  in  the  ROCC  central  computer. 
Installing  the  ATS  at  the  radar  head  will  decentralize  the  computer 
processing.  The  ATS  will  discriminate  between  clutter  and  target 
data,  dump  the  clutter  data,  and  pass  only  the  target  data  to  the 
central  computer  for  further  processing.  The  main  benefits  to  the 
central  computer  are  reductions  in  the  amount  of  processing 
required  and  the  potential  for  a  computer  overload  that  arbitrarily 
dumps  data. 

The  reduced  computer  workload  will  enable  the  installation 
of  additional  radars.  This  is  important  because  PACAF  wants  to 
expand  the  air  training  ranges  in  Alaska.  The  eruption  of  Mount 
Pinatubo  and  subsequent  loss  of  a  major  air  training  exercise  in  the 
Philippines  called  Cope  Thunder  required  more  training,  and  thus  an 
expansion  of  the  air  training  ranges,  in  Alaska.  As  a  result, 
additional  radars  are  needed  to  provide  coverage  in  these  new 
training  areas.  However,  the  additional  data  from  these  sites  could 
easily  create  an  overload  condition  in  the  computer.  Dumping  data 
during  a  major  exercise  would  seriously  jeopardize  the  safety  of  the 
entire  mission. 

The  life  of  the  central  computer  will  be  prolonged  because 
the  amount  of  data  the  central  computer  has  to  process  will  be 
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eased.    Thus,  the  costly  acquisition  of  a  new  generation  ROCC 
computer  can  be  delayed. 

3.      Mistakes  (Experimental  Design) 

The  details  of  the  experimental  design  should  be  included  in  the  plan.  It 
should  explain  the  experimental  setup  in  detail,  including  the  physical  setup,  who 
the  test  subjects  or  experimental  units  are,  what  special  equipment  is  required,  and 
give  the  trial  schedule.  The  hypotheses  that  are  to  be  examined  should  be  stated 
specifically,  as  should  any  assumptions  that  will  be  made.  The  statistical  design  of 
the  experiment  should  be  explained,  along  with  definitions  of  measures  that  will 
be  employed  and  instrumentation  that  will  be  used  to  collect  the  data.  Provisions 
should  be  made  for  testing  the  setup  and  design,  and  for  the  trials. 

While  the  Introduction  section  of  the  Plan  fails  to  provide  the  necessary 
focus  for  the  experiment,  the  Experimental  Design  section  suffers  from  a  lack  of 
detail.  The  closest  the  Plan  comes  to  an  experimental  design  is  the  Procedures 
Section,  a  subsection  of  the  Demonstration  Objectives. 

The  Procedures  Section  attempts  to  cover  too  many  details  in  a  single 
page  of  material.  Of  the  five  statements  listed  on  page  4-2,  only  three  can  be 
considered  procedures.  These  three  statements,  taken  from  the  Plan,  are  quoted 
below: 

Three  radar  sites  (Cape  Romanzof,  Kotzebue,  and  Tatalina)  will  be  used  to 
collect  data  to  satisfy  objectives  one  and  two  (reduce  demands  on  the  FYQ- 
93  ROCC  central  computer,  and  radar  enhancement).  These  sites  were 
chosen  because  they  possess  the  latest  clutter  reducing  software  (i.e.  ECP 
175  and  the  Multiscan  Detection  Processing  (MDP)).  At  different  times  each 
day,  data  will  be  collected  from  each  site  for  a  period  of  105  minutes.  Each 
fifteen  minutes,  within  the  data  collection  window,  the  radar  will  be  placed 
into  a  different  configuration  (e.g.  ECP  175  enabled  and  MDP  disabled).  Data 
will  be  collected  via  data  reduction,  later  collated  on  the  Site  Data  Sheets 
(figure  1 )  and  analyzed. 
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Data  will  be  collected  using  low  level  flights  (if  available)  in  the  ACMI 
range.  This  data  will  be  used  to  assess  the  ATS'ss  ability  to  enhance  the 
probability  of  detection  (Pd)  of  the  FPS  -117  radar. 

A  tracking  test  will  be  used  to  meet  the  third  demonstration  objective 
(increase  probability  of  detection  by  reducing  human  variability  in  the 
detection/tracking  process).  A  operator  will  be  selected  at  random  and  given 
an  area  of  responsibility  (AOR)  to  initiate  and  maintain  track  continuity  on  all 
tracks  in  the  AOR.  The  operator  will  operate  with  and  without  the  benefits  of 
the  ATS  and  all  track  histories  will  be  recorded  via  data  reduction.  A 
comparison  will  be  made  of  the  data  collected  from  the  data  reduction 
products  to  determine  whether  or  not  an  operator  will  realize  an  increased 
probability  of  detection  by  reducing  human  variability  in  the 
detection/tracking  process.  Detailed  instructions  will  be  provided  to  each 
surveillance  operator  by  the  project  coordinator-thus  insuring  a  degree  of 
procedural  standardization. 

These  three  statements  seem  to  try  to  map  back  to  the  original  objectives 
that  were  not  clearly  defined  in  the  first  place.  But,  the  result  is  unsuccessful,  and 
the  reader  is  left  confused  as  to  the  specific  outcome  the  procedures  are  trying  to 
accomplish.  The  lack  of  clearly  defined  objectives  has  an  adverse  effect  on  how 
the  experiment  is  to  be  conducted.  As  a  result,  the  reader  comes  away  without  a 
clear  idea  as  to  what  problem  the  experiment  is  trying  to  solve. 

Another  problem  is  that  there  are  no  procedures  describing  how  the 
objective  of  "Reducing  the  demands  on  the  FYQ-93  ROCC  central  computer..." 
will  be  accomplished.  Is  this  objective  no  longer  important? 

The  arbitrary  presentation  of  procedural  details  further  illustrates  a  lack 
of  focus  in  the  Plan.  Paragraph  two  states  that  three  sites  were  selected  based 
upon  the  installation  of  certain  ECPs.  Why  does  it  matter  which  ECP  is  installed 
at  each  radar?  Should  not  one  of  the  main  procedures  of  the  experiment  be  to 
disable  all  of  the  clutter  reducing  ECPs?  If  this  is  the  case,  then  it  should  not 
matter  what  kind  of  ECP  is  installed  at  the  radar.  The  procedure  suggests  that  the 
experiment  is  designed  to  compare  the  various  ECPs  located  at  different  radar 
sites  with  the  ATS.  This  procedure  does  not  map  back  to  any  of  the  objectives. 
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To  further  illustrate  the  arbitrary  nature  of  the  procedures,  paragraph 
two  states  that  data  will  be  collected  for  a  period  of  105  minutes,  and  that  every 
15  minutes  the  radar  will  change  to  a  different  ECP  configuration.  There  is  no 
explanation  given  why  a  105  minute  data  collection  period  was  chosen. 
Furthermore,  manipulating  the  configuration  of  the  ECPs  is  unnecessary.  The 
details  mentioned  in  paragraph  two  only  distract  and  confuse  the  reader. 

Finally,  paragraph  two  mentions  that  data  will  be  collected  via  data 
reduction  and  then  analyzed.  Several  questions  are  not  addressed.  First,  what 
kind  of  data  is  being  collected?  How  will  the  researcher  know  if  this  data  will 
provide  answers  to  the  research  questions  if  it  is  not  clear  what  the  questions  are? 
What  will  the  data  measure?  How  will  the  data  be  collected?  The  data  will  be 
collected  by  data  reduction,  what  does  that  mean?  What  instrumentation  will  be 
required  to  collect  this  data?  What  is  the  plan  for  analyzing  the  data  after  it  is 
collected?  All  of  these  questions,  and  more,  need  to  be  included  as  details  in  the 
design  of  the  experiment. 

Paragraph  three  states:  "Data  will  be  collected  using  low  level  flights  (if 
available)  in  the  ACMI  range.  This  data  will  be  used  to  assess  the  [ATS's]  ability 
to  enhance  the  probability  of  detection  (Pd)  of  the  FPS-1 17  radar."  This  is  what 
the  experiment  is  suppose  to  be  about,  but,  no  details  are  given  as  to  how  this  will 
be  carried  out.  The  lack  of  detail  makes  the  reader  wonder  whether  or  not  the 
experiment  will  be  conducted  "on  the  fly." 

Paragraph  four  is  a  procedure  for  another  experiment  that  would 
determine  whether  or  not  the  ATS  will  increase  the  probability  of  detection  by 
reducing  the  human  variability  in  the  detection/tracking  process.  This  procedure 
fails  to  define  details  of  how  the  experiment  will  be  executed.  For  example:  the 
procedure  states:  "A[n]  operator  will  be  selected  at  random  and  given  an  area  of 
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responsibility  (AOR)  to  initiate  and  maintain  track  continuity  on  all  tracks  in  the 
AOR."  What  will  his  performance  be  compared  to?  What  kind  of  operator  will  be 
selected?  Background  information  is  required  here  because  there  are  different 
types  and  capabilities  of  ROCC  operators.  Furthermore,  how  will  the  operators 
be  selected;  based  upon  what  criteria?  Again,  the  issue  of  what  measures  will  be 
used  are  not  addressed.  How  will  the  data  be  collected?  What  instrumentation 
will  be  used  to  collect  the  data?  What  is  the  plan  for  analyzing  the  data?  How 
does  this  experiment  provide  the  evidence  necessary  to  determine  the  capabilities 
of  the  ATS?  Does  this  experiment  focus  on  the  primary  objective?  Or,  is  this 
experiment  an  example  of  how  an  experiment  can  get  off  track  because  of  a  lack 
of  clear  objectives? 

4.      Solutions  (Experimental  Design) 

The  experimental  design  problems  are  solved  by  utilizing  the  concepts 
found  in  the  model  and  the  paradigm.  According  to  the  paradigm,  "the 
Experimental  Design  is  comprehensive  but  executable  if  it  tests  all  the  pertinent 
hypotheses  with  measurable  data  by  a  practical  analysis  plan."  In  order  to 
accomplish  this  task,  the  model  must  be  used.  The  revised  Experimental  Design 
section  that  follows  applies  both  the  model  and  the  paradigm  and  is  a  direct 
extension  of  the  revised  Introduction  section. 
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H.  REVISED  EXPERIMENTAL  DESIGN 
A.  Setup: 

1.  Physical 

The  physical  setup  is  the  crux  of  the  entire  experiment.  The 
demonstration  would  not  be  possible  if  not  for  the  flexibility  of  the 
ROCC  system.  Figure  4  depicts  the  actual  physical  interface 
between  the  ROCC  and  the  ATS  equipment.  The  following  is  a 
discussion  of  the  ROCC- ATS  interface. 

The  digital  switch  provides  the  interconnection  between  the 
communication  channels  from  the  radar  sites  and  the  central 
computer.  The  radar  data  at  the  digital  switch  has  not  been 
processed  by  the  ROCC  computer.  Each  radar  sends  its  data  to  the 
digital  switch  at  the  ROCC  via  three  2.4  kbps  communications 
channels.  These  channels  feed  directly  into  three  ports  associated 
with  each  radar  at  the  digital  switch.  Two  ports,  designated  A  and 
B,  are  normally  interconnected  with  the  primary  and  standby  central 
computers.  The  third  port,  designated  C,  is  available  for  recording 
data  on  a  noninterference  basis.  This  is  the  location  of  the  ROCC- 
ATS  interface.  At  the  same  time,  the  cable  from  port  B  of  the  Digital 
Switch  to  the  central  computer  will  be  disconnected  at  the 
computer,  and  the  output  from  the  ATS  will  be  connected  to  the 
standby  computer  at  its  input  channel  (Figure  4). 

2.  Factors: 

There  are  three  experimental  factors:  radar  sites  at  three 
levels;  Litton  ATS  at  two  levels;  and  analysis  equipment  at  two 
levels.  Three  radar  sites  have  been  chosen:  Kotzebue,  Cape 
Romanzof,  and  Tatalina  (Figure  2).  The  ATS  radar  processing 
hardware/software  is  either  on  or  off.  Finally,  the  equipment  being 
used  to  analyze  the  data  is  either  third  party,  Air  Force  approved, 
test  and  evaluation  equipment  or  the  Litton  Data  Systems  analysis 
program  associated  with  the  ATS. 


30 


3.  Special  Equipment: 

a.  The  ATS  consists  of  all  of  the  interface  hardware  as  well  as 
the  software  to  conduct  the  data  collection  and  analysis.  All  will  be 
provided  by  Litton  Data  Systems. 

b.  A  neutral  third  party  will  be  involved  to  provide  balance 
and  legitimacy  to  the  experiment.  This  third  party  (i.e.,  84th  Radar 
Evaluation  Squadron)  must  conduct  the  experiment  in  parallel  with 
Litton.  They  must  interface  at  the  same  location,  with  the  same  site, 
and  at  the  same  time.  They  will  collect  the  data,  analyze  it,  and 
compare  their  results  with  Litton's  experimental  results. 

Aircraft  will  be  dedicated  to  fly  certain  profiles  within  the 
coverage  limits  of  the  selected  radar. 

4.  Schedule  of  Trials: 

The  schedule  of  trials  will  not  be  formally  addressed  in  this 
sample  plan.  The  driving  factor  for  the  schedule  of  trials  will  be  the 
availability  of  the  aircraft  needed  to  fly  the  flight  profiles.  Making 
up  a  schedule  of  trials  does  not  provide  sufficient  lessons  learned. 

B.  Hypotheses: 

la.  There  is  no  difference  between  the  amount  of  data 
forwarded  by  the  fully  sensitized  radar  and  the  radar  equipped  with 
the  ATS. 

lb.  The  alternative:  There  is  a  difference  between  the 
amount  of  data  forwarded  by  the  fully  sensitized  radar  and  the  radar 
equipped  with  the  ATS. 

2a.  There  is  no  difference  between  the  number  of  targets 
reported  by  the  ROCC  computer  combined  with  the  fully  sensitized 
radar  and  the  radar  equipped  with  an  ATS. 
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2b.  There  is  a  difference  between  the  number  of  targets 
reported  by  the  ROCC  computer  combined  with  the  fully  sensitized 
radar  and  the  radar  equipped  with  an  ATS. 

3a.  There  is  no  difference  between  the  third  party  analysis 
results  and  the  Litton  Data  Systems  analysis  results. 

3b.  There  is  a  difference  between  the  third  party  analysis 
results  and  the  Litton  Data  Systems  analysis  results. 

The  first  two  hypotheses  deal  with  the  issues  of  quantity  and 
quality  of  the  data  the  ATS  produces.  Does  the  ATS  make  a 
difference  in  the  amount  of  data  being  sent  back  to  the  ROCC 
central  computer?  And,  does  the  ATS  discriminate  targets  from 
clutter  without  eliminating  valid  air  targets? 

The  third  hypothesis  is  designed  to  validate  the  Litton 
demonstration  by  determining  whether  or  not  the  data  is  consistent 
with  that  from  an  established  test  and  evaluation  system.  The  1  ltn 
ACW  staff  must  be  sure  that  the  ATS  works  as  claimed  and  that  the 
analysis  tools  that  are  part  of  the  equipment  package  portray  the 
data  in  an  accurate  manner. 

C.  Assumptions: 

1 .  Litton's  interface  hardware  is  compatible  with  the  ROCC 
equipment. 

2.  The  software  will  provide  the  type  of  analysis  needed  to 
answer  the  research  questions. 

3.  The  weather  patterns  and  clutter  density  at  the  three  sites 
are  representative  of  the  (AOR)  overall. 

4.  Both  Litton  and  the  third  party  Air  Force  evaluation 
equipment  is  capable  of  counting  and  keeping  track  of  the  number 
of  targets  received  from  the  radar. 
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Note:  This  assumption  is  based  on  the  claims  of  Litton  that  its 
software  has  the  capability  to  compile  this  type  of  information  [Ref . 
3].  Additionally,  since  the  ROCC  computer,  a  1970s  version 
computer,  has  the  capability  to  count  the  number  of  tracks  in  the 
system,  and  is  certified,  it  is  assumed  that  the  evaluation  equipment 
exists  within  the  Air  Force,  or  else  the  ROCC  computer  could  not 
have  been  certified  in  the  first  place. 

D.  Statistical  Design  of  Experiment 

The  experiment  is  a  3  X  2  X  2  factorial  single  blind 
experiment.  This  means  that  one  factor  has  three  levels  and  the 
other  two  factors  each  have  two  levels.  The  first  factor  has  three 
levels  because  of  the  selection  of  three  different  radar  sites.  The 
radar  sites  associated  with  these  levels  are:  Kotzebue,  Cape 
Romanzof,  and  Tatalina.  This  factor  corresponds  to  hypotheses  one 
and  two  stated  above.  The  processing  equipment  is  the  second 
factor.  The  two  levels  are  whether  or  not  the  radar  is  utilizing  the 
ATS.  This  factor  also  corresponds  to  hypotheses  one  and  two.  The 
third  factor  is  analysis  equipment.  The  two  levels  for  this  factor  are 
the  third  party  operationally  approved  equipment  and  the  Litton 
analysis  equipment.  This  factor  corresponds  to  hypothesis  number 
three.  The  purpose  of  designing  a  single  blind  experiment  is  to 
eliminate  experimenter  bias  and  to  serve  as  a  means  to  evaluate 
whether  or  not  the  ATS  is  eliminating  valid  air  targets. 

E.  Measures 

The  experiment  is  designed  to  evaluate  the  data  forwarded 
by  the  radar.  The  data  will  be  collected  and  reduced  into  two 
categories.  The  first  category  deals  with  the  quantity  of  data  and 
the  second  deals  with  the  quality  of  data  (target  elimination).  The 
categories  of  data  are  listed  below. 

Category  one:  Quantity.  The  total  amount  of  data  processed 
and  sent  by  the  radar  to  the  ROCC,  measured  in  bits  per  second. 
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Category  two:  Quality.  The  total  number  of  targets  being 
sent  by  the  radar  to  the  ROCC,  measured  by  the  total  number  of 
targets  being  sent  by  the  ROCC  computer  to  the  operator. 

F.  Instrumentation 

The  instrumentation  is  the  equipment  used  by  Litton  and 
the  third  party  Air  Force  evaluation  team  to  collect  and  analyze  the 
data. 

G.  Testing  &  Pilot  Trials 

Testing  will  be  done  to  ensure  the  ATS  and  third  party 
equipment  can  be  connected  to  the  ROCC  digital  switch. 
Additional  testing  will  be  done  to  ensure  the  commander  that  the 
experiment  will  not  interfere  with  the  daily  operations  of  the  ROCC. 

After  the  interfaces  are  made,  pilot  trials  will  be  run  to  ensure 
that  the  data  being  collected  is  the  correct  data  to  test  the 
hypotheses.  The  researchers  will  use  the  pilot  testing  to  practice  the 
experimental  procedures.  For  planning  purposes,  time  should  be 
allotted  in  the  master  schedule  for  the  pilot  testing. 

5.      Mistakes  (Data  Description) 

A  description  of  the  data  should  also  be  included  in  the  plan.  This 
section  of  the  plan  should  contain  an  example  of  the  raw  data,  (the  entire  set  of 
collected  data  should  appear  in  an  appendix)  problems  experienced  with 
collecting  the  data  should  be  addressed,  and  the  data  should  be  presented  in  an 
understandable  and  easy  to  read  format  (i.e.,  tables  or  spreadsheets).  Any 
methods  used  to  reduce  the  data,  or  other  manipulations,  should  be  explained. 

The  way  the  Plan  describes  the  data  is  another  contributing  factor  to  its 
overall  weakness.  The  Plan  describes  the  data  in  a  subsection  entitled  Data 
Products.     The  opening  statement:    "Data  products  required  to  ensure  accurate 
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analysis."  implies  that  the  data  collected  will  answer  the  research  questions  being 
asked.  The  analysis  of  the  Experimental  Design  section  revealed  that  the  plan 
does  not  mention  how  the  data  will  be  collected.  The  lack  of  measures,  coupled 
with  the  fact  that  no  testing  and  pilot  trials  are  called  for  in  the  plan,  leads  one  to 
believe  that  no  one  would  know  if  the  wrong  data  had  been  collected  until  after 
the  analysis  had  begun.  This  is  too  late  to  learn  that  the  data  collected  will  not 
answer  the  research  questions. 

Another  problem  with  the  Data  Products  section  is  that  it  states  the  data 
is  to  be  collected  from  the  Aircraft  Combat  Maneuvering  and  Instrumentation 
Center  computer  (ACMI)  and  the  ROCC.  However,  there  is  no  description  of  the 
data  that  will  be  collected  from  these  two  points.  The  Plan  also  mentions  that  a 
log  will  be  maintained  at  the  Air  Surveillance  Technician  position.  A  log  may  be 
useful  to  help  reconstruct  the  experiment,  but  it  should  be  kept  by  a  person 
directly  involved  with  the  experiment.  Furthermore,  a  log  of  this  nature  does  not 
provide  any  real  data  for  analysis. 
6.     Solution  (Data  Description) 

As  with  the  other  sections  of  the  Plan,  the  revised  Data  Description 
Section  that  follows  provides  a  possible  solution  based  upon  the  lessons  learned 
from  the  mistakes  of  the  original  plan  and  utilizing  the  model  and  paradigm. 

m.  REVISED  DATA  DESCRIPTION 

Table  1  presents  the  data  coding  scheme.  The  table  depicts 
the  three  factors  (Radar  Sites,  Analysis  Equipment,  Processing 
Equipment)  along  with  their  associated  levels.  The  number  in 
parenthesis  is  the  code  assigned  to  the  particular  level.  The  coding 
scheme  will  become  apparent  when  the  table  listing  the  different 
combinations  of  trials  is  presented  below. 
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TABLE   1     DATA   CODING   SCHEME 


Radar  Sites  Analysis    Equipment  Processing    Equipment 

Kotzebue  ( 1 )  Third  Party  ( 1 )  Non  ATS  ( 1 ) 

CapeRomanzoff(2)  Litton  (2)  ATS  (2) 
Tatalina  (3) 


The  data  is  divided  into  two  categories. 

Data:  Total  amount  of  data  received  from  the  radar  (X  in  bits 
per  second) 

Total  number  of  tracks  (Y) 

The  data  is  divided  into  two  categories  to  aid  in  the  analysis, 
but  it  will  all  be  collected  at  once  for  each  trial.  The  computer 
software  of  the  evaluation  equipment  will  reduce  the  data  into  the 
X  and  Y  categories  designated  above. 

Table  2  depicts  the  combinations  of  levels  of  the  factors. 
There  are  twelve  combinations  of  factor  levels,  found  by  multiplying 
one  factor  with  three  levels  and  two  factors  each  with  two  levels. 
Each  of  the  trials  will  be  replicated  thirty  times  in  order  to  assume 
the  data  approximates  a  normal  distribution. 
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TABLE  2     TRIALS 


Trial 

# 

Radar 

Sites 

Analysis 

Equip. 

Processing  Equip.  Data  (X)  Data  (Y) 

1 

1 

1 

1 

2 

1 

1 

2 

3 

1 

2 

1 

4 

1 

2 

2 

5 

2 

1 

1 

6 

2 

1  • 

2 

7 

2 

2 

1 

8 

2 

2 

2 

9 

3 

1 

1 

10 

3 

1 

2 

11 

3 

2 

1 

12 

3 

2 

2 

There  are  a  several  steps  in  the  data  collection  process.  The  first 
step  is  to  collect  all  of  the  data  the  radar  is  sending  to  the  ROCC. 
The  total  amount  of  data  will  be  used  by  the  analysis  to  answer 
hypotheses  one  and  three.  Target  counts  will  then  be  compiled 
using  the  software  capabilities  of  the  computers.  The  target  counts 
will  be  used  to  answer  hypothesis  number  two.  Additionally,  the 
target  count  data  can  be  used  to  provide  supplementary  information 
concerning  hypotheses  numbers  three. 

7.     Mistakes  (Analysis  Plan) 

A  good  plan  must  state  how  the  analysis  will  be  performed  (the  details 
of  the  analysis  belong  in  the  report).  The  analysis  plan  must  also  state  the 
statistical  tests  that  will  be  used  to  analyze  the  data. 

The  Plan  does  not  address  how  the  data  will  be  analyzed,  only  that  once 
the  data  is  collected  it  will  be  analyzed. 
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8.     Solutions  (Analysis  Plan) 

The  analysis  section  is  the  final  section  of  the  revised  plan.  Like  the 
sections  before,  the  mistakes  made  in  the  original  plan  are  overcome  by  using  the 
model  and  the  paradigm  as  a  guide. 

IV.  REVISED  ANALYSIS  PLAN 

Balanced  Analysis  of  Variance  (ANOVA)  will  be  used  to 
analyze  the  data.  The  results  will  determine  the  outcome  of  the 
hypotheses  tests.  As  indicated  in  the  data  section,  there  are  two 
parts  to  the  plan.  The  first  section  will  use  the  balanced  ANOVA  to 
analyze  the  total  amount  of  data.  Thirty  replications  of  each  of  the 
twelve  trials  will  be  made.  This  will  be  done  to  provide  normality. 
The  second  phase  of  the  analysis  will  also  use  balanced  ANOVA, 
this  time  to  analyze  the  thirty  replications  of  target  counts. 

B.    REPORT  ANALYSIS 

The  same  analytical  tools  used  to  evaluate  the  ATS  Plan  will  be  used  to 
analyze  the  Report.  The  analysis  will  compare  the  information  in  the  Report  to 
what  should  be  included  in  a  report  based  upon  the  model  and  the  paradigm.  As 
with  the  analysis  of  the  Plan,  mistakes  are  addressed  and  solutions  are  presented. 
However,  a  revised  Report  is  not  presented  because  data  crucial  to  the  analysis 
was  not  collected  or  is  not  available  for  analysis.  Since  the  entire  report  revolves 
around  the  Analysis  Section,  and  data  is  not  available  to  be  analyzed,  merely 
restating  what  has  already  been  said  in  the  revised  Plan  and  leaving  the  Analysis 
Section  out  would  have  little  value. 

The  formal  title  of  the  original  report  is:  LRR  ENHANCEMENTS  STAFF 
STUDY.  This  is  the  final  report  to  the  1 1th  Air  Force  Commander  recommending 
a  course  of  action  based  upon  the  outcome  of  the  RDMS  demonstrations.    It  is 
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interesting  to  note  that  only  one  demonstration  plan  but  two  demonstrations  are 
addressed  in  the  final  report. 

Two  demonstrations  were  conducted  by  the  1 1  ACW.  First,  Litton  Data 
Systems  demonstrated  the  Advanced  Tracking  System  (ATS).  The  planning  for 
this  demonstration  took  an  entire  year  and  culminated  in  the  execution  of  the 
ATS  Demonstration  Plan.  After  the  ATS  demonstration  and  analysis  were 
complete,  Sensis  Corporation  hearing  that  their  competitor  Litton  Data  Systems 
had  conducted  a  demonstration  in  Alaska,  contacted  the  1 1  ACW  expressing 
their  desire  to  demonstrate  their  equipment.  The  1 1  ACW,  concerned  with  the 
legal  consequences  of  not  letting  Sensis  Corporation  demonstrate  their 
equipment,  allowed  a  Sensis  demonstration.  Because  of  the  short  time  interval 
between  the  Litton  and  Sensis  demonstrations,  a  plan  could  not  be  written, 
approved,  and  disseminated.  As  result,  a  Sensis  Plan  does  not  exist.  The  Report  is 
the  first  time  the  Sensis  demonstration  is  mentioned. 

1.     Mistakes  (Introduction) 

An  Introduction  Section  of  an  experimental  report  is  similar  to  that 
already  discussed  for  an  experimental  plan.  This  section  should  introduce  the 
researchers  involved  with  the  experiment,  contain  a  well  defined  purpose, 
including  a  statement  of  the  real  world  problem  and  specific  questions  the 
experiment  seeks  to  answer,  the  experimental  approach,  and  anticipated  results,  as 
well  as  the  scope  of  the  experiment.  In  addition,  the  Introduction  Section  of  the 
report  should  also  contain  any  additional  information  that  may  have  occurred 
during  the  course  of  the  experiment. 

The  Report  fails  the  same  the  way  the  Plan  fails.  It  does  not  meet  the 
requirements  of  either  the  model  or  the  paradigm.  The  Report  does  not  contain  a 
clearly  defined  Introduction  Section.   The  analysis  of  the  Plan  demonstrated  that 
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if  the  Introduction  Section  failed  to  state  a  clear  objective  the  rest  of  the  Plan 
suffered  because  it  is  too  easy  to  get  side  tracked  from  the  original  purpose  of  the 
experiment. 

There  is  one  overarching  problem  with  the  Report.  There  is  a  shift  in  the 
emphasis  from  what  the  Plan  set  out  to  accomplish  and  what  the  Report  is  trying 
to  accomplish.  The  first  sentence  in  the  Report  reveals  the  discontinuity  between 
the  Plan  and  the  Report.  The  first  sentence  states:  "The  1  lm  AF  needs  to  acquire 
a  system  that  will  integrate  existing  systems  i.e.,  3DS  [Digital  Data  Display 
System],  AICU  [Advanced  Interface  Control  Unit],  and  ACMI  [Air  Combat 
Maneuvering  and  Instrumentation],  into  a  cohesive,  efficient  system  providing 
unseen  benefits  to  the  Alaska  2000+  Range  Improvements  Project"  (see  Glossary, 
Appendix  A,  for  definition  of  terms).  Remember,  the  Plan  was  to  focus  on 
validating  the  claims  of  Litton  Data  Systems  Advanced  Tracking  System 
capabilities.  However,  the  first  statement  in  the  Report  clearly  indicates  a  shift  to 
a  different  emphasis. 

The  next  area  the  Report  addresses  is:  Factors  Bearing  on  the  Problem. 
The  first  factor  (paragraph  a)  states  the  requirement  for  integrating  another  type 
of  radar  into  the  ROCC.  This  is  valid  requirement.  However,  isn't  this 
requirement  a  benefit?  If  the  Plan  and  the  execution  of  the  experiment  had 
focused  on  validating  the  ATS's  capabilities,  and  the  report  presented  evidence  to 
support  these  capabilities,  then  the  RDMS  could  be  considered  a  viable  solution. 
The  next  three  factors,  also  taken  from  this  section,  address  integration  issues 
(paragraph  b,c,and  d)  and  appear  below. 

b.  Currently,  no  capability  exists  to  automatically  insert  flight 
data  on  military  aircraft  into  3DS.  The  NORAD  Software  Support 
Facility  (NSSF)  is  working  on  CSCP  [Computer  Software  Change 
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Proposal]  #  4569  which  would  allow  the  automatic  tell  of  military 
aircraft.  Hence,  creating  the  pathway  needed  to  get  data  on  military 
aircraft  into  3DS.  The  solution  the  NSSF  is  trying  to  achieve  is  a 
major  software  change:  no  solution  is  on  the  horizon. 

c.  Track  data  on  aircraft  not  equipped  with  ACMI  pods,  but 
still  players  in  large  force  exercises  (LFEs),  are  not  entered  into  the 
ACMI. 

d.  Radar  data  on  aircraft  not  equipped  with  ACMI  pods  are 
detected  by  LRRs  surrounding  the  range;  however,  this  information 
is  only  sent  to  the  ROCC.  The  idea  here  is  to  provide  data  on 
exercise  participants,  not  capable  of  being  tracked  by  ACMI,  to  the 
ACMI  via  the  LRR-ROCC/AICU-ACMI  interface  (see  attach.  1). 
The  results:  a  more  complete,  accurate  air  picture,  providing  an 
added  measure  of  flight  safety  both  to  the  military  and  civil  sectors. 

The  statements  that  there  exists  an  inability  to  automatically  insert  flight 
plan  data  as  well  as  track  data  on  aircraft  not  equipped  with  ACMI  pods  into  the 
3DS  is  an  attempt  by  the  1 1  ACW  to  provide  evidence  to  support  the  RDMS  as  a 
solution.  A  requirement  to  automatically  fuse  data  derived  from  outside  C3 
systems  may  exist,  but  the  experiment  was  not  designed  to  support  this  question. 
Furthermore,  no  evidence  exits  in  the  Report  to  support  this  requirement. 

The  integration  issues  also  address  the  important  safety  role  the  radars 
play  in  keeping  track  of  aircraft  not  equipped  with  ACMI  pods.  Aircraft  not 
equipped  with  ACMI  pods  are  not  tracked  by  the  ACMI  and  the  radars  provide 
the  only  measure  of  safety.  While  this  is  true,  and  there  is  a  requirement  to 
integrate  this  data,  the  report  claims  an  added  measure  of  safety  it  can  not  support 
with  data  obtained  by  the  experiment.  The  experiment  was  not  designed  to 
support  a  flight  safety  issue.  Therefore,  conclusions  about  flight  safety  can  not  be 
extrapolated  from  the  experiment  that  was  conducted. 


41 


The  next  factor  addressed  (paragraph  e)  is  the  communications  interface 
between  the  ROCC  and  the  radar. 

e.  The  communication  interface  between  the  LRR  and  the 
ROCC  needs  to  be  made  more  efficient.  At  present  massive 
amounts  of  radar  data,  most  of  it  useless  clutter  is  being  bulk 
encrypted  and  sent  over  the  communications  circuits  to  the  ROCC 
(see  attach.  2).  Sending  only  true  target  information,  filtered  from 
the  clutter,  will  reduce  the  burden  on  the  interface,  thereby  making 
this  interface  more  efficient. 

Again,  this  a  valid  requirement,  but  one  that  is  not  addressed  by  the 
current  Plan.  In  order  to  make  any  claims  about  how  much  burden  would  be 
reduced  by  an  RDMS,  experiments  have  to  be  designed  with  this  question  in 
mind. 

The  report  states  (paragraph  f)  that  one  of  the  factors  is:  "Too  much 
useless  data  is  being  sent  to  the  ROCC  computer  for  processing.  Sending  only 
data  which  contains  true  target  information  will  inherently  make  the  ROCC 
computer  more  efficient."  How  can  this  statement  be  verified  from  the  design  of 
the  experiment  put  forth  in  the  Plan?  Again,  a  separate  experiment  must  be 
conducted  if  the  1 1  ACW  wants  to  verify  these  claims. 

Human  variability  is  addressed  in  paragraph  (g).  The  Report  says:  "A 
human  variability  in  the  target  detection  and  tracking  process  exits.  "  There  is  no 
denying  that  variability  exists  between  individuals  in  the  tracking  process. 
Furthermore,  it  would  seem  likely  that  if  the  RDMS  lived  up  to  its  capability  the 
variability  in  the  tracking  process  could  be  reduced.  However,  the  Report  sites 
one  of  its  attachments  (see  attachment  3  in  Appendix  D)  as  supporting  evidence 
to  this  claim.  This  attachment  is  a  view  graph  of  a  radar  scope  demonstrating  the 
difference  in  the  amount  of  data  between  an  ATS  equipped  radar  and  a  non-ATS 
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equipped  radar  as  seen  from  an  operator  perspective.  This  view  graph  does  not, 
however,  support  the  claim  that  variability  in  the  tracking  process  can  be 
reduced.  Visually  looking  at  the  differences  does  not  tell  the  whole  story.  The 
view  graph  does  not  provide  insight  as  to  how  the  data  is  being  processed  by  the 
ATS  or  whether  the  ATS  is  eliminating  targets  of  interest  (the  primary  objective  of 
the  Plan).  A  statistical  analysis  of  the  raw  data  needs  to  be  accomplished 
according  to  a  design  like  that  laid  out  in  the  new  Plan. 

The  final  factor  the  Report  mentions  is  the  need  to  integrate  all  of  the 
stovepipe  C2  systems  in  the  theater  (paragraph  h).  The  reports  says:  "Systems, 
i.e.,  AICU,  ACMI,  CTIS  [Commander's  Theater  Information  System]/3DS,  need  to 
be  integrated  more  efficiently.  Information  available  to  one  system  can  be  made 
available  to  another  system.  For  example:  track  data  the  AICU  receives  from  an 
LRR  can  be  made  available  to  both  3DS  and  ACMI."  Again,  this  is  clear 
evidence  of  the  shift  in  emphasis  between  the  Plan  and  the  Report.  The  emphasis 
has  shifted  from  validating  the  ATS's  capabilities  to  using  the  Range 
Improvements  Project  requirements  as  a  means  of  justifying  the  procurement  of 
the  RDMS. 

The  fifth  and  final  section  of  the  Report  is  the  Discussion  Section.  The 
material  mentioned  in  this  section  could  either  be  totally  eliminated  from  the 
report  or  included  as  part  of  the  Introduction  Section,  providing  background. 

The  general  impression  of  the  Discussion  Section  is  that  the  RDMS  is 
the  "catch  all"  single  solution  to  all  of  the  Range  Improvements  Project's 
requirements.  The  introduction  paragraph  states:  "A  solution  will  be  offered, 
tested  against  criteria,  and  shown  how  this  solution  will  solve  the  problem."  The 
report  argues  that  if  the  RDMS  can  hold  up  to  the  scrutiny  of  criteria  that  are 
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established  for  the  first  time  in  the  Report,  then  the  RDMS  must  be  the  a  valid 
solution. 

The  entire  discussion  appears  to  be  a  sales  pitch  to  the  decision  makers 
on  the  importance  of  the  RDMS  to  the  Range  Improvements  Project.  The  Report 
essentially  ignores  the  original  purpose  of  the  demonstration,  as  stated  in  the  Plan 
or  otherwise,  and  argues  that  the  RDMS  is  the  only  solution  because  it  meets  all 
of  the  criteria  the  Report  established.  The  Discussion  Section  makes  no  attempt 
to  address  the  issue  of  whether  or  not  the  claims  made  by  the  vendors  about  their 
equipment  are  valid.  The  only  things  offered  are  excerpts  from  the  contractors 
final  reports  on  how  great  their  equipment  is  and  all  the  reasons  the  lltn  AF 
should  buy  their  equipment.  There  is  no  independent,  unbiased,  scientific 
analysis  of  the  data.  The  only  analysis  is  from  the  vendors,  using  their  equipment 
and  their  analysis  tools. 

The  1 1 tn  ACW  is  asked  to  have  faith  that  the  analysis  conducted  by  the 
vendors,  using  their  own  equipment,  is  reliable.  The  demonstration  started  out 
with  the  stated  purpose  of  determining  the  validity  of  the  ATS's  capabilities  and 
ended  up  using  the  ATS  as  a  solution  to  an  integration  problem.  This  drift  can  be 
traced  back  to  the  lack  of  focus  of  the  Plan,  which  did  not  provide  a  "road  map" 
that  the  11th  ACW  staff  could  follow.  This  lack  of  focus  lead  to  a  Plan  and 
Report  that  lacked  credibility  and  do  not  provide  concrete  answers  for  decision 
makers.  This  is  probably  why  the  PACAF  has  yet  to  acquire  a  RDMS  for  the 
Alaskan  radars. 

2.      Solution  (Introduction) 

The  Introduction  section  in  the  final  report  provides  information  to  the 
reader  of  the  renort  who  has  not  read  or  had  access  to  the  Plan.  Therefore, 
everything  mentioned  in  the  Introduction  Section  of  the  revised  Plan  should  be 
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included  in  the  final  report,  plus  any  additional  information  revealed  during  the 
experiment. 

During  the  time  of  the  demonstrations,  the  Alaska  2000+  Range 
Improvements  Project  became  a  prime  interest  of  the  1  ltn  AF  commander.  The 
1  ltn  ACW  saw  the  Range  Improvements  Project  as  a  funding  vehicle  for  their 
RDMS  project.  The  background  politics  explain  the  discontinuity  between  the 
Report  and  the  Plan;  however,  this  is  not  an  excuse  for  allowing  a  shift  in  the 
problem  emphasis.  The  RDMS  project  can  only  suffer.  When  the  focus  shifts 
back  to  the  RDMS,  questions  will  be  asked  about  the  validity  of  its  claimed 
capabilities.  If  both  the  Plan  and  Report  focused  on  the  questions  of  validating 
RDMS  capabilities,  then  the  llm  ACW  would  have  strong  supporting  evidence 
for  piggybacking  on  the  Range  Improvements  Project. 
3.      Mistakes  (Experimental  Design) 

Like  the  Introduction  Section,  the  Experimental  Design  Section  shares 
similarities  between  the  experimental  plan  and  report;  the  difference  is  that  greater 
detail  must  be  included  in  the  report.  It  should  explain  the  experimental  setup 
including  the  physical  setup,  who  the  test  subjects  or  experimental  units  are,  what 
special  equipment  is  required,  and  give  the  trial  schedule.  The  hypotheses  that 
are  to  be  examined  should  be  stated  specifically,  as  should  any  assumptions  that 
will  be  made.  The  statistical  design  of  the  experiment  should  be  explained,  along 
with  definitions  of  measures  that  will  be  employed  and  instrumentation  that  will 
be  used  to  collect  the  data.  Provisions  should  be  made  for  testing  the  setup  and 
design,  and  for  the  trials. 

The  Report  fails  to  adequately  address  the  experimental  design.  The 
report  contains  Assumptions  and  Criteria  Sections  that  could  be  considered 
elements  of  the  experimental  design  section.    The  assumptions  deal  only  with 
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integration.  The  assumptions  are:  "Industry  has  the  capability  to  solve  the 
integration  requirements  we  have  established  during  this  study.  The  AICU  will 
be  able  to  receive  track  messages  from  the  RDMS  and  route  these  messages  back 
and  forth  between  the  3DS  and  ACMI."  However,  assumptions  used  to  analyze 
the  data  collected  from  the  experiment  are  not  mentioned. 

The  Criteria  Section  deals  only  with  requirements  that  have  to  be  met  in 
order  to  provide  an  adequate  solution  to  the  interface  problems.  Each  of  the  five 
criteria  are  biased  towards  the  RDMS  as  a  solution.  The  report  lists  the  following 
as  criteria: 

a.  The  solution  must  be  able  to  reduce  the  burden  on  the  LRR-ROCC 
interface. 

b.  The  data  that's  being  output  is  in  a  format  that  can  be  used  by  the 
ROCC  computer  as  well  as  the  AICU. 

c.  The  solution  must  be  able  to  integrate  and  interface  the  AN/TPS-63 
radar  data  in  the  correct  format  into  the  ROCC  where  it  will  be  routed  via  the 
AICU  to  the  3DS  and  ACMI. 

d.  The  system  must  have  an  open  architecture  design,  being  able  to  be 
incorporated  into  future  systems  (ROMS/RAMMS). 

e.  Versatile  user  friendly  workstations,  providing  self  diagnostics  and  real 
time  radar  status  monitoring,  as  well  as  GCI  control. 

Clearly  each  of  the  criteria  takes  advantage  of  the  proposed  benefits 
touted  by  the  vendors  of  the  RDMS  equipment.  Once  again,  the  political 
pressures  are  at  work:  If  the  RDMS  could  meet  all  of  the  criteria,  how  could  the 
decision  makers  ignore  the  RDMS  and  not  allow  funding  to  a  system  that  directly 
impacts  the  Range  Improvements  Project? 
4.     Solutions  (Experimental  Design) 

The  report  should  address  the  experimental  design  in  same  manner  as 
the  Plan.  The  experimental  design  section  tells  the  reader  how  the  experiment 
was  carried  out.    No  details  should  be  left  out.   The  report  must  provide  details 
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concerning  the  setup  of  the  experiment,  hypotheses  must  be  clearly  stated  (these 
are  the  specific  research  questions),  assumptions  made  must  be  listed,  the 
statistical  design  of  the  experiment  must  be  presented,  the  issue  of  what  kind  of 
data  was  collected,  and  how  it  is  to  be  collected,  must  not  be  overlooked,  and 
testing  and  pilot  trials  have  to  be  mentioned. 

5.  Mistakes  (Data  Description) 

This  section  of  the  plan  should  contain  an  example  of  the  raw  data,  (the 
entire  set  of  collected  data  should  appear  in  an  appendix)  problems  experienced 
with  collecting  the  data  should  be  addressed,  and  the  data  should  be  presented  in 
an  understandable  and  easy  to  read  format  (i.e.,  tables  or  spreadsheets).  Any 
methods  used  to  reduce  the  data,  or  other  manipulations,  should  be  explained. 

The  Report  does  not  even  mention  the  data.  There  is  little  doubt  as  to 
why  the  incorrect  data  was  collected  because  neither  the  Plan  nor  the  Report  was 
concerned  about  data  collection. 

6.  Solution  (Data  Description) 

All  of  the  details  described  in  the  revised  Plan  need  to  be  included  in  the 
final  report.  As  a  matter  of  fact,  the  report  should  contain  even  more  detail  than 
the  plan.  The  report  should  contain  an  example  of  the  raw  data,  and  problems  the 
data  should  be  addressed.  All  of  the  data  collected  should  be  presented  in  an 
appendix  or  in  an  archive. 

Readers  may  want  to  replicate  the  experiment  to  verify  the  results 
claimed  in  the  report.  The  data  coding  scheme  (an  example  appears  in  the  revised 
Plan)  must  be  included.  This  will  allow  the  reader  to  decipher  the  way  the 
experimenter  coded  the  data,  and  it  will  aid  in  understanding  the  analysis.  If 
possible,  and  for  the  sake  of  clarity,  the  data  should  be  presented  in  a  tabular 
form.   If  the  data  is  manipulated  or  reduced  in  any  way,  this  has  to  be  addressed. 
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The  bottom  line:    The  data  description  must  be  detailed  and  accurate  enough  so 
that  the  experiment  can  be  replicated  and  the  results  verified. 

7.  Mistakes  (Analysis) 

The  Analysis  section  is  probably  the  most  important  section  of  the 
report.  This  section  should  contain  a  summary  of  the  analysis  plan  that  was 
presented  in  the  experimental  plan,  a  detailed  methodology  of  how  the  analysis 
was  conducted,  the  results  of  the  analysis,  and  any  additional  assumptions  that 
were  required. 

The  Report  lacks  any  significant  analysis  section.  No  statistical  analysis 
of  the  data  is  accomplished.  The  view  graph  attachments  at  the  end  of  the  report 
(see  Appendix  D)  are  the  only  things  presented  as  the  analysis.  Basically,  the  1 1 
ACW  presented  the  findings  of  the  vendors.  No  attempt  was  made,  nor  was  any 
data  provided,  to  verify  the  claims  made  by  the  vendors.  The  problems  with  the 
analysis  stem  all  the  way  back  to  the  lack  of  focus  and  clearly  defined  objectives 
in  the  Plan  and  the  resulting  inability  to  maintain  focus  from  the  inception  of  the 
Plan  through  its  execution. 

8.  Solution  (Analysis) 

Data  that  can  provide  the  answers  to  specific  research  questions, 
generated  from  well  defined  objectives,  is  a  key  piece  of  the  solution  to  the 
analysis  failure.  There  are  several  elements  that  are  essential  to  the  analysis 
section.  The  analysis  should  contain  a  summary  of  the  analysis  plan  presented  in 
the  initial  experiment  plan.  The  methodology  used  to  collect  the  data  and  how 
that  data  was  analyzed  must  be  detailed,  and  any  additional  assumptions  that  the 
experimenter  made  have  to  be  addressed.  The  results  of  the  analysis  are  the  heart 
of  the  entire  experimental  process.    They  results  allow  conclusions  to  be  drawn 
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and   recommendations   to   be   made   that  are   objectively   based   on   actual 
experimental  outcomes. 

9.      Mistakes  (Conclusions) 

The  Conclusion  Section  is  unique  in  that  it  only  appears  in  the 
experimental  report.  This  section  presents  the  results  of  the  hypotheses  that  were 
listed  in  the  plan,  as  well  as  any  other  interpretations  drawn  from  the  analysis  of 
the  data.  These  results  are  then  related  to  the  real  world  problem  the  experiment 
set  out  to  answer.  An  overall  summary  of  the  experiment  should  also  be 
presented  in  this  section. 

The  Report  states:  "...  demonstrations  conducted  by  Litton  Data 
Systems,  Sensis  Corp.,  and  GE  Syracuse,  have  clearly  shown  that  industry  possess 
the  capability  to  meet  our  established  criteria.  A  RDMS  at  the  LRR  would 
integrate  radar  data  from  both  the  AN/FPS-117  and  AN/TPS-63  [the  proposed 
Westinghouse  radar  that  would  be  acquired  to  enhance  radar  coverage  in  the 
expanded  training  areas]  radars  into  the  ROCC,  3DS,  and  ACMI  while  at  the 
same  time  increasing  the  efficiency  of  the  entire  Alaska  2000+  Range  network." 
First,  the  1 1  ACW  had  little  or  no  involvement  with  the  GE  demonstration,  so  it 
will  not  be  addressed  in  this  analysis.  Second,  the  motivation  behind  the  report 
was  to  show  how  the  RDMS  equipment  could  be  used  to  integrate  the  various 
in-theater  C2  systems.  Clearly  a  different  objective  than  what  was  initially 
proposed:  To  verify  the  claims  made  by  the  vendors  about  their  equipment 
capabilities.  Since  the  report  argues  for  the  integration  benefits  of  the  RDMS 
equipment,  the  analysis  presented  in  the  attachments  is  arguing  in  favor  of  the 
capabilities.  In  short,  the  conclusions  are  not  valid.  There  is  no  evidence  to 
support  the  conclusions  being  drawn. 
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10.  Solution  (Conclusions) 

The  main  ingredient  to  solving  the  problem  with  the  conclusions  is 
maintaining  a  constant  focus  on  the  objectives  of  the  experiment  throughout  the 
Plan  and  Report.  One  approach  to  achieving  this  focus  is  the  use  of  the  model 
and  paradigm.  The  model  contains  four  elements  in  the  Conclusion  Section.  The 
first  element  is  the  hypotheses  results,  along  with  the  interpretations  of  these 
results.  The  second  element  is  any  other  interpretations.  Third,  and  probably  the 
most  important,  the  real  world  meaning  of  the  results.  Finally,  an  experimental 
summary  should  be  drawn.  According  to  the  paradigm,  the  conclusions  are 
"...supportable  if  they  are  valid  results  of  an  analysis  that  follows  logically  from 
the  study." 

11.  Mistakes  (Recommendations) 

The  report  should  finish  with  a  Recommendations  Section.  This  section 
should  contain  a  discussion  of  potential  changes  to  the  experiment  along  with 
recommendations  for  further  work  in  this  area. 

The  Report  does  not  contain  the  elements  described  above,  instead  the 
Report  makes  a  recommendation  in  favor  of  the  RDMS.  Based  upon  all  of  the 
analysis  of  this  Report,  there  is  no  evidence  to  support  this  recommendation. 

12.  Solutions  (Recommendations) 

The  Recommendations  Section  of  the  report,  in  addition  to  offering 
recommendations,  should  contain  sections  dealing  with  changes  to  be  made  to 
the  experiment.  Additionally,  the  recommendations  section  should  address  any 
requirement  for  further  work  in  this  area. 

13.  General  Issues 

Finally,  there  are  two  general  issues  concerning  the  overall  quality  of  the 
experiment.  First,  is  the  issue  of  simplicity.  According  to  the  paradigm,  the  report 
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is  "...simple  if  it  is  short  and  illustrative,  has  a  minimum  number  of  assumptions, 
alternatives,  hypotheses,  and  measures,  plus  easily  understood  analysis  and 
conclusions."  The  Report  fails  to  meet  the  definition  of  simplicity.  While  it  is 
short,  the  Report  fails  to  illustrate  how  the  experiment  was  carried  out.  The 
assumptions  are  not  representative  of  an  experiment  initially  designed  to  verify 
the  capabilities  of  the  ATS.  Instead  of  listing  specific  hypotheses,  integration 
criteria  are  listed.  Neither  measures  or  an  analysis  is  addressed  by  the  plan.  A 
marginal  attempt  at  an  analysis  is  made  using  view  graphs.  Finally,  the 
conclusions  stated  apply  to  benefits  of  integration  and  not  about  RDMS 
capabilities. 

The  second  issue  addresses  the  consistency  of  the  report.  "The  study  is 
consistent  if  there  is  compatibility  in  scope  and  detail  at  each  step.  In  addition, 
direct  linkage  between  the  steps  must  maintain  continuity  with  the  objective." 
The  shift  in  the  objectives  between  the  Plan  and  Report  is  evidence  the  study  is 
not  consistent.  There  is  no  continuity  between  the  Plan  and  the  Report. 

C.  CONCLUSIONS 

The  ATS  Demonstration  Plan  was  analyzed  using  the  Experiment  Plan  and 
Report  Formats  as  a  model  of  how  a  plan  is  organized.  The  Evaluation  of  Studies 
paradigm  provided  definition  to  the  organizational  model.  Highlighting  the 
mistakes  of  the  Plan  was  only  part  of  the  analysis.  The  next  step  was  to  take  the 
lessons  learned  from  the  Plan  and  apply  them  to  a  new  hypothetical  plan. 

The  analysis  of  the  Report  (known  as  the  LRR  STAFF  STUDY)  focused  on 
the  analysis  and  recommendations.  Like  the  analysis  of  the  Plan,  the  Report 
analysis  utilized  the  model  and  paradigm  as  evaluation  tools.  The  Plan  and  Report 
are  similar  because  most  of  the  material  covered  in  the  Plan  is  summarized  in  the 
Report.      Unlike  the  analysis  of  the  Plan,  the  analysis  of  the  Report  does  not 
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provide  a  new  Report  because  data  is  missing  from  both  Litton  and  Sensis.  The 
missing  data  can  not  simply  be  made  up  in  order  to  complete  an  analysis 
consistent  with  the  new  Plan. 

The  analysis  reveals  many  mistakes  in  both  the  Plan  and  Report.  However, 
many  lessons  can  be  learned  from  this  experience.  These  lessons  have  been 
applied  in  the  creation  of  a  new  Plan.  Furthermore,  generic  lessons,  applicable  to 
experiments  and  experimenters,  are  found  in  Chapter  IV. 
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IV.     LESSSONS  LEARNED 

Specific  lessons  can  be  learned  by  studying  the  Plan  and  Report  described  in 
this  thesis.  The  analysis  chapter  illustrated  many  mistakes  that  can  be  made 
during  an  experiment.  This  chapter  takes  the  specific  mistakes  made  during  the 
RDMS  experiments  and  creates  a  list  of  lessons  learned.  Additionally,  the  lessons 
the  author  (who  was  a  major  player  in  the  original  experiment)  learned  are  listed 
and  are  applicable  to  experiments  in  general  Hopefully,  these  lessons  learned  may 
serve  as  a  guide  to  staff  officers  that  have  little  experience  in  conducting 
experiments  but  are  tasked  to  conduct  an  experiment  in  an  operational 
environment. 

A.    SPECIFIC  LESSONS  LEARNED 

Experimental  plans  and  reports  have  specific  roles.  The  plan  describes  in 
detail  how  the  experiment  will  be  executed.  The  report  describes  the  execution 
of  the  experiment,  providing  an  analysis,  drawing  conclusions  and  making 
recommendations.  The  lessons  described  in  this  section  all  relate  to  specific 
mistakes  identified  in  the  Plan  and  the  Report. 

1.      Clearly  Defined  Objectives 

Clearly  defined  objectives  are  a  must.  They  provide  the  foundation 
upon  which  the  entire  experiment  is  based.  For  example,  the  objectives  defined  in 
the  experimental  plan  provide  the  focus  for  the  entire  experiment.  Without  this 
focus  the  researchers  may  find  it  very  difficult,  if  not  impossible,  to  carry  out  a 
successful  experiment. 
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2.  State  the  Purpose 

State  the  purpose  of  the  experiment,  preferably  in  an  Introduction 
Section,  both  in  the  plan  and  in  the  final  report.  The  purpose  should  describe  the 
real  world  problem  the  experiment  is  trying  to  solve  along  with  specific  questions 
the  experiment  seeks  to  answer.  This  is  important  because  it  helps  to  maintain  the 
focus  for  the  experiment.  After  this  groundwork  is  laid,  a  description  of  the 
approach  the  researchers  will  use  to  try  and  solve  the  problem  must  be  addressed. 
Finally,  to  help  build  a  common  mental  picture  of  what  the  experiment  hopes  to 
achieve,  there  should  be  a  statement  about  the  anticipated  results  of  the 
experiment. 

3.  Define  the  Scope  of  the  Experiment 

The  importance  of  this  experiment,  why  the  experiment  is  being 
conducted,  and  what  impact  will  it  have,  constraints  placed  on  the  conduct  of  the 
experiment,  and  boundaries  on  the  applicability  of  the  results,  should  be  spelled 
out  in  a  section  that  describes  the  scope  of  the  experiment. 

4.  Provide  a  Detailed  Experimental  Design 

The  experimental  design  is  the  heart  of  the  plan.  This  section  should 
provide  the  details  of  how  the  experiment  is  to  be  executed.  It  should  be  detailed 
enough  so  that  when  the  experiment  is  verified  it  can  be  replicated. 

5.  Provide  a  Data  Description 

What  kind  of  data  is  going  to  be  collected  and  used  for  analysis?  Will 
an  analysis  of  the  data  provide  answers  to  the  questions  being  asked?  These 
questions  must  be  answered  prior  to  the  start  of  the  actual  experimental  trials. 
The  researcher  must  know  what  kind  of  data  is  being  collected  and  whether  or 
not  it  will  be  useful. 
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6.  Prepare  an  Analysis  Plan 

After  the  data  is  collected,  it  must  be  analyzed.  To  prevent  bias  in  the 
analysis,  and  to  ensure  that  the  right  data  is  collected,  the  researcher  must  have  a 
plan  of  how  the  data  will  be  analyzed.  While  the  analysis  plan  is  stated  in  the 
experimental  plan,  the  actual  analysis  and  results  are  the  crux  of  the  final  report. 

7.  Conduct  Testing  and  Pilot  Trials 

A  preferred  method  of  determining  whether  or  not  the  correct  data  will 
be  collected  during  the  experiment  is  by  conducting  testing  and  pilot  trials  prior 
to  the  actual  execution  of  the  experiment.  Testing  and  pilot  trials  prior  to  the  start 
of  the  lltn  ACW  experiments  would  have  determined  that  the  data  being 
collected  was  incomplete  and  would  not  adequately  provide  for  an  analysis  that 
would  yield  insights  as  to  the  capabilities  of  the  RDMSs. 

8.  State  Supportable  Conclusions 

The  results  of  the  analysis  should  answer  the  specific  questions  being 
asked.  The  conclusions  must  be  supported  by  the  results  of  the  analysis. 

9.  Make  Sound  Recommendations  Based  Upon  Analysis  of  the  Data 
The  recommendations  that  are  made  should  be  supported  by  data 

collected  through  the  execution  of  the  experiment  and  not  by  circumstantial 
evidence. 

B.    GENERALIZED  LESSONS  LEARNED 

The  generalized  lessons  learned  are  subjective  lessons  the  author  developed 
during  the  course  of  this  thesis.  These  lessons,  hopefully,  will  serve  as  a  guide  to 
those  staff  officers  with  little  background  in  experimentation,  but  are  nonetheless 
called  upon  to  conduct  experiments  in  an  operational  environment. 
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1.  Ask  Questions 

Can  experiments  be  conducted  in  an  operational  environment? 
What  will  the  operational  impact  be  if  the  experiment  is  conducted? 
Will  the  operational  mission  adversely  affect  the  experiment? 
Will  experiments  conducted  in  the  operational  arena  yield  useful  results? 
Should  commander  involve  the  staff  to  conduct  these  experiments? 
Is  the  staff  knowledgeable  enough  to  undertake  this  task?  Or  will  outside 
assistance  be  required? 

2.  Gather  Knowledge  Base 

Ensure  knowledge  exits  within  the  staff  to  conduct  such  an  experiment. 
If  necessary  seek  help  from  other  sources.  Maybe  even  have  some  other  unit  or 
agency  come  in  and  conduct  the  experiment.  This  will  alleviate  any  bias  the 
home  staff  may  have  towards  the  experiment.  Two  views  existed:  one  favoring 
the  ATS  equipment,  and  one  wanting  no  part  of  the  equipment.  Interestingly,  the 
lines  of  opposing  opinions  were  drawn  between  operations  and  maintenance 
staffs.  The  operations  staff  wanted  to  see  the  ATS  equipment  incorporated  while 
the  maintenance  staff  wanted  to  continue  on  with  the  current  ECP  fixes. 

3.  Be  Open  Minded,  Yet  Remain  Cynical 

Don't  get  locked  into  one  solution  and  fail  to  seek  other  viable  options. 
Be  cynical,  ask  hard  questions.  Don't  be  impressed  with  "bells  and  whistles." 
Seek  proof  of  claims  from  independent  sources. 

4.  A  Good  Plan  is  a  Must 

A  good  plan  must  have  clearly  defined  objectives.  These  objectives  are 
then  broken  down  into  questions  the  plan  will  be  designed  to  answer. 
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5.  Keep  Research  Questions  Simple 

Keep  research  questions  simple  and  in  line  with  operational  constraints. 
Trying  to  solve  complicated  research  questions  in  a  constrained  operational 
environment  may  prove  to  be  too  difficult  and  best  left  to  the  laboratory.  Break 
the  complicated  research  question  down  into  smaller  pieces. 

6.  Pilot  Trials  are  Essential 

Pilot  trials  ensure  the  data  collected  will  provide  answers  to  the 
questions  being  asked.  Ensure  the  data  is  replaceable. 

7.  Keep  Detailed  Notes 

Trying  to  remember  of  the  details  of  an  experiment  is  an  impossible  task 
unless  detailed  notes  are  kept. 

8.  Do  Not  be  Afraid  to  Modify  the  Experiment 

If  the  pilot  trials  indicate  flaws  with  the  experiment,  change  whatever 
you  need  to  change.  Objectives  may  have  to  be  modified.  Whatever  the  case,  do 
not  be  afraid  to  admit  mistakes.  The  important  thing  is  to  correct  the  mistakes 
before  the  actual  trials  begin.  Time  and  money  are  precious  commodities.  If 
mistakes  remain  undiscovered  or  overlooked,  rerunning  the  entire  experiment 
may  not  be  feasible. 

9.  Use  the  Right  Statistical  Tool  for  the  Right  Job 

Using  certain  statistical  tools  requires  different  assumptions  to  be  made. 
Using  the  wrong  tool  seriously  undermines  the  conclusions  drawn  from  the 
analysis. 

10.  Report  Results  in  a  Clear,  Concise,  and  Simple  Format 

Do  not  attempt  to  give  a  technical  brief  to  a  commander  and  his  staff. 
You  could  quickly  find  yourself  out  in  the  hall.   If,  by  chance  you  are  allowed  to 
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proceed,  you  run  the  risk  of  creating  a  great  deal  of  confusion  and  seriously 
jeopardizing  an  credibility  of  your  office. 

11.  Present  Only  the  Facts,  Don't  Speculate 

When  asked  to  report  the  results  of  your  analysis  report  only  the  results. 
If  you  do  not  have  the  answer  say  so  and  do  not  interject  personal  opinions. 
Doing  so  only  clouds  the  results  of  the  experiment. 

12.  Be  Careful  About  Extrapolating  Results  to  Other  Situations 

This  lesson  is  a  corollary  of  the  above  lesson.  There  is  an  overwhelming 
urge  to  extrapolate  the  results  of  experiments  to  other  areas  of  interest.  Care  must 
be  taken.  The  results  of  the  experiment  are  for  a  certain  set  of  conditions. 
Extrapolating  results  of  an  experiment  to  a  different  set  of  conditions  without 
further  research  is  not  wise.  In  other  words,  stay  within  the  scope  of  the 
experiment. 

13.  Ensure  Consistency  Between  the  Plan  and  the  Report 

The  conclusions  presented  in  the  final  report  should  be  consistent  with 
the  primary  objective  stated  originally  in  the  plan.  If,  for  some  reason  the 
objectives  of  the  experiment  change,  then  the  plan  should  reflect  the  changes 
prior  to  conducting  the  experiment. 
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V.     CONCLUSIONS 

The  data  collected  could  not  provide  the  answers  to  the  primary  questions, 
and  that  analysis  of  the  Plan  and  Report  revealed  several  weaknesses  which  no 
doubt  contributed  to  this  failure.  A  summary  of  the  deficiencies  is  given  in  the 
following  paragraphs. 

The  overarching  problem  with  the  Plan  was  that  it  failed  to  clearly  define  the 
objectives  of  the  experiment.  This  resulted  in  a  lack  of  a  clearly  defined  purpose. 
As  a  result,  the  Plan  constantly  strayed  from  the  primary  objective:  verifying  the 
ATS's  capabilities. 

The  Plan  also  failed  to  provide  a  detailed  description  of  the  experimental 
design.  As  it  is  presented,  the  Plan  does  not  describe  how  the  experiment  will  be 
conducted  to  verify  the  ATS's  capabilities.  This  resulted  in  procedures  calling  for 
experiments  that  did  not  relate  to  the  primary  objective. 

The  lack  of  a  clearly  defined  data  description  resulted  in  an  inability  to 
conduct  a  thorough  analysis.  The  Plan  states  that  data  will  be  collected  and  then 
analyzed.  However,  there  is  no  description  of  the  data,  how  it  will  be  collected,  or 
any  testing  done  to  determine  whether  or  not  the  data  is  the  correct  data  needed 
for  the  analysis.  The  analysis  could  not  be  conducted  because  insufficient  data 
was  collected. 

The  Plan  also  lacks  an  analysis  section.  There  is  no  mention  as  to  how  the 
data  will  be  analyzed.  The  result  is  seen  in  the  Report  where  the  analysis  uses 
subjective  arguments  to  support  certain  criteria  and  view  graphs  showing  the 
benefits  in  reducing  the  amount  of  data  being  sent  by  the  radars. 
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The  lack  of  clear  objectives  also  plagued  the  Report.  As  a  result,  the  entire 
emphasis  of  the  Report  shifted  from  verifying  RDMS  equipment  capabilities  to 
integrating  other  in-theater  C3  systems. 

The  Report  lacked  any  significant  analysis.  Instead  of  analyzing  collected 
data  to  test  the  validity  of  the  ATS,  a  set  of  criteria,  not  previously  discussed,  was 
presented.  The  analysis  consisted  of  examining  whether  or  not  the  RDMS 
equipment  would  meet  the  criteria.  As  a  result,  little  if  any,  information  could  be 
gleaned  from  the  analysis. 

The  conclusions  presented  in  the  Report  were  not  supported  by  a  statistical 
analysis  from  data  collected  during  the  experiment.  Instead,  the  Report  presents 
view  graphs  providing  a  visual  representation  of  the  RDMS  equipment 
capabilities.  The  result  is  that  there  are  still  questions  about  the  validity  of  the 
RDMS  and  its  capabilities.  The  conclusions  seemed  to  be  based  upon 
circumstantial  evidence  not  on  evidence  provided  by  the  analysis  data  collected 
during  the  experiment. 

The  final  result  of  the  Plan  and  Report  is  a  positive  endorsement  of  the 
RDMS  as  a  solution  to  the  radar  data  management  problems  found  in  Alaska. 
However,  based  upon  the  analysis,  the  recommendations  made  in  the  Report  are 
of  doubtful  validity. 

The  model  used  in  this  thesis  is  an  accepted  evaluation  tool.  There  is  no 
requirement  that  all  experimental  plans  and  reports  follow  this  model;  but,  it  is  the 
opinion  of  this  author  that  the  Plan  and  Report  failed  partly  as  a  result  of  not 
following  a  model  such  as  the  one  taught  at  the  Naval  Postgraduate  School. 
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VI.     SUMMARY  AND  RECOMMENDATIONS 

The  purpose  of  this  thesis  was  to  analyze  an  actual  experimental  plan  and 
report  and  provide  lessons  learned.  The  entire  experimental  process,  starting  with 
the  plan  development  and  proceeding  through  the  publication  of  the  final  report, 
lasted  approximately  two  years.  The  motivation  behind  this  thesis  was  to 
illustrate  the  difficulties  involved  with  experimentation.  Compounding  these 
difficulties  is  the  fact  that  this  author,  who  was  in  charge  of  the  experiment, 
lacked  the  proper  training  needed  to  carry  out  an  experiment  successfully. 
However,  after  attending  the  Naval  Postgraduate  School  (NPS)  and  studying  the 
correct  way  to  conduct  an  experiment,  this  author  decided  to  use  the  lessons 
learned  at  NPS  to  analyze  what  was  done  two  years  before. 

Chapter  II  provided  background  stating  the  motivating  factors  behind  the 
experiment.  Chapter  III  is  the  heart  of  the  thesis.  This  chapter  analyzed  the  ATS 
Demonstration  Plan  and  the  LRR  Enhancements  Staff  Study.  Mistakes  were 
pointed  out  and  solutions  were  offered  based  upon  the  Experiment  Plan  and 
Report  Format  as  well  as  the  Evaluation  of  Studies  paradigm  presented  in  the 
Joint  Command  and  Control  and  Communications  curriculum  offered  at  the  Naval 
Postgraduate  School.  Chapter  IV  provided  the  lessons  learned  from  the 
experiment.  The  purpose  of  this  chapter  was  to  provide  a  guide  to  those  staff 
officers  who  have  little  experience  in  experimental  design  and  analysis  but  are 
nonetheless  tasked  to  conduct  an  experiment  in  an  operational  environment. 
Hopefully,  the  lessons  learned  may  serve  as  a  guide  so  that  these  staff  officers  can 
provide  reliable  recommendations  to  decision  makers. 

Further  work  needs  to  be  done  to  determine  the  validity  of  the  RDMS. 
Experiments  need  to  be  conducted  that  determine  whether  or  not  the  RDMS 
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device  is  eliminating  data  on  targets  vital  to  air  defense  and  air  combat  training. 
After  this  is  determined,  questions  of  integrating  this  system  into  current  C3 
architectures  can  be  addressed. 
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APPENDIX   A 

GLOSSARY 
ACMI 

ACMI  stands  for  Aircraft  Combat  Maneuvering  and  Instrumentation.    The 
ACMI  was  initially  designed  as  a  training  device  for  fighter  pilots.    The  ACMI 
provides  a  three  dimensional  view  of  aircraft  in  relation  to  its  environment 
providing  lessons  learned  for  mistakes  made  in  flight. 
AICU 

AICU  stands  for  Advanced  Interface  Control  Unit.  This  system  is  designed  as 
a  translator  and  router  of  data.  The  purpose  of  the  AICU  is  to  serve  as  a  main 
point  where  data  from  different  sensors  can  be  interchanged.  Data  from  different 
sensors  utilizing  one  type  of  protocol  can  be  exchanged  with  data  of  other 
sensors  utilizing  a  different  protocol.  For  example,  data  from  a  Radar 
Management  device  located  at  a  radar  can  be  sent  to  the  AICU  translated  into  an 
appropriate  protocol  so  the  data  can  be  displayed  on  the  3DS  as  well  as  on  the 
radar  consoles  of  the  ROCC. 
CTIS/3DS 

CTIS/3DS  stands  for  Commander's  Theater  Information  System/Digital  Data 
Display  System.  CTIS  is  a  network  of  computers  designed  to  collect  and  fuse 
data.  The  fused  data  is  then  displayed  using  a  large  screen  display.  The  computer 
hardware  and  software  required  to  project  the  fused  data  is  called  the  Digital  Data 
Display  Device  (3DS). 
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GCI 

Ground  Control  Intercept.    Air  weapons  control  officers  station  in  a  ground 
based  radar  unit  with  the  task  of  controlling  aircraft. 
LRR 

Long  range  radars.     In  Alaska  the  LRR  is  the  AN/FPS-117  Minimally 
Attended  Radar. 
Plot 

Data  necessary  to  describe  a  point  in  space.   A  two  dimensional  view  (X  and 
Y  axes) 
Target 

Consists  of  more  than  one  plot.   Gives  the  third  dimension.  Associated  data: 
Heading,  Speed,  Altitude.    A  direction  can  be  determined  from  a  series  of  plots 
called  a  target. 
Track 

A  track  is  a  target  with  alphanumeric  data  (symbology)  attached.     The 
symbology  can  either  be  manually  or  computer  derived.    Tracks  are  established 
by  a  radar  operator  who  determines  whether  or  not  a  target  represents  a  valid 
aircraft.  The  operator  has  a  direct  involvement  in  the  decision  making  process. 
ROMS/RAMMS 

Remote  operational  maintenance  system  designed  to  provide  long  distance 
status  monitoring  of  the  radar. 
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APPENDIX  B 
ADVANCED  TRACKING  SYSTEM  DEMONSTRATION  PLAN 
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11th  TACTICAL  CONTROL  WING  (PACAF) 
ELMENDORF  AFB    AK     99506-2270 
1  October  1991 


ADVANCED  TRACKING  SYSTEM 
DEMONSTRATION  PLAN 


PACIFIC  AIR  FORCES 

1 1TH  TACTICAL  CONTROL  WING 

ELMENDORF  AFB    ALASKA     99506-2270 
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FROM:  11TCW/DO 

6900  9th  ST  STE  301 

ELMENDORF  AFB  AK  99506-2270 

SUBJECT:  Advanced  Tracking  System  Demonstration  Plan 

TO:  SEE  DISTRIBUTION 

1 .  Forwarded  herewith  is  the  Advanced  Tracking  System  Demonstration  Plan,  1 
Oct  91 .  This  demonstration  is  being  conducted  at  the  direction  of  1 1  AF/CC. 

2.  This  plan  outlines  those  actions  necessary  to  demonstrate  the  capabilities  and 
utility  of  the  Advanced  Tracking  System  in  the  Alaskan  Theater. 

3.  This  plan  is  designed  to  demonstrate  the  operational  capabilities  of  the 
Advanced  Tracking  System  as  part  of  the  current  Alaskan  Surveillance  and 
Command  and  Control  System.  Of  particular  interest  will  be  the  Automatic 
Tracking  System's  ability  to  reduce  demands  on  the  ROCCs  central  computer  ( 
processing  time  and  memory  space),  restore  the  FPS-117,  Long  Range  Radar, 
performance  to  its  original  operational  specifications  for  target  detection,  and  to 
reduce  human  variability  in  the  detection  and  tracking  process.  The 
demonstration  will  take  place  from  7  Oct  91  to  18  Oct  91 .  The  equipment  will  be 
located  in  the  ROCC  in  order  to  take  advantage  of  the  existing  communications 
circuits  feeding  radar  data  from  the  Long  Range  Radars  (LRRs)  to  the  ROCC 
computer.  Locating  the  ATS  in  the  ROCC  will  allow  access  to  any  radar  without 
the  logistics  of  transporting  people  and  equipment  to  remote  sites  and  make  the 
equipment  available  for  maximum  visibility. 


GEORGE  A.  PAHLS,  Colonel,  USAF  1  Atch 

Deputy  Commander  for  Operations  Advanced  Tracking  System 

Demonstration  Plan 
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FLAM  SUMMARY 


1.  PURPOSE:  To  outline  responsibilities  and  procedures  for  the  Advanced 
Tracking  System  Demonstration  of  7  Oct- 1 8  Oct  91 . 

2.  CONDITIONS  FOR  IMPLEMENTATION:  This  project  plan  will  be  the  basis 
of  the  Advanced  Tracking  System  Demonstration.  It  will  be  implemented  at  the 
direction  of  the  1 1  TCW/DO. 

3.  OPERATIONS  TO  BE  CONDUCTED: 

a.  Forces  Assigned.   All  tasked  units  will  respond  when  this  plan  is  executed 
unless  specifically  excused  by  the  implementing  authority. 

b.  Supporting  Plans.  Not  required. 

4.  OPERATIONS  SECURITY  (OPSEC):  In  the  execution  of  this  plan,  OPSEC 
must  be  a  matter  of  continuing  concern  in  order  to  minimize  disclosure  of 
sensitive  information.  Since  operations  may  reveal  limitations  of  the  Automatic 
Tracking  System,  the  AN/FPS-1 17  Long  Range  Radar,  and  the  ROCC  AN/FYQ- 
93  computer,  activity  must  be  conducted  so  as  to  preclude  inadvertent  disclosure 
of  sensitive  and/or  classified  information. 

5.  COMMAND  RELATIONSHIPS:  Normal. 

6.  LOGISTICAL  APPRAISAL:  All  logistical  matters  envisioned  are  feasible. 
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IPI[JIRIP(Q)§IE 


1.  The  stated  purpose  of  the  demonstration  is  determine  whether  or  not  the  ATS 
can  be  used  in  the  Alaskan  Theater  to  improve  radar  performance  and  make 
recommendations  concerning  the  desirability  of  incorporation  the  ATS  in  the 
Alaskan  Surveillance  and  Command  and  Control  System.  The  primary  objective 
is  to  collect  sufficient  data  to  determine  the  ATS'  capability  for  substantially 
improving  the  probability  of  detecting  targets  (at  lower  altitudes  and  clutter 
environment).  The  ATS,  by  sending  only  true  target  data  to  the  ROCC  computer, 
could  reduce  the  amount  of  data  the  computer  has  to  process.  A  significant 
reduction  of  data  will  reduce  the  demands  on  computer  processing  time  and 
memory/storage  space;  both  of  which  are  nearly  saturated. 


1-1 
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1.  In  order  to  manage  the  demonstration,  an  Advanced  Tracking  System  Project 
Management  Team  was  established.  The  demonstration  team  leader  is  1 1 
TCW/DOX;  other  team  members  are  listed  in  paragraph  3. 

2.  The  charter  of  the  project  management  team  is  to:  set  the  demonstration 
objectives;  develop  evaluation  criteria;  conduct  and  observe  demonstrations, 
record  data  on  the  results  of  the  demonstration;  evaluate  the  potential 
application  of  the  ATS;  and  publish  a  project  report. 

3.  The  following  agencies  are  participating  or  providing  support  for  the  ATS 
demonstration: 

a.  1 1  TCW/DOXXQ --  Capt  Pierce 

b.  11TCW/DOP       -  SSgtMyher 

c.  1 1  TCW/LGOR    --  MSgt  DeLuca 

d.  11TCW/LGKC    --  MSgtShuler 

e.  11TCW/LGKM    --  TSgtCushman 

f .  744  ADS/DOO      --  Lt  McNeil 

g.  21  st  TFW/DOW  --  Capt  Hill 


2-1 


72 


STOTTIEM  IDIESCMIFiriKON 


1.  The  AN/GYQ  -51,  Advanced  Tracking  System,  manufactured  by  Litton  Data 
Systems  is  a  state  of  the  art  tracking  system,  providing  sensor  interoperability 
between  centralized,  decentralized,  and  autonomous  air  surveillance  systems.  The 
ATS  is  a  front  end  processor  located  at  the  radar,  intelligently  filtering  true  targets 
from  clutter  and  sending  only  actual  track  data  to  the  command  center  for 
display.  This  capability  allows  larger  volumes  of  actual  track  data  to  be 
transmitted  over  communications  channels  without  resorting  to  desensitizing  or 
blanking  of  the  radar. 


2.  The  ATS  functions  by  detecting,  establishing,  and  displaying  tracks  without 
operator  intervention.  The  ATS  accomplishes  these  functions  through  a  four  step 
process: 

a.  considers  all  radar  returns  via  scan  to  scan  processing; 

b.  identifies  true  targets; 

c.  establishes  tracks  using  true  targets;  and 

d.  maintains  tracking  continuity  using  predictive  logic. 


3-1 


73 


DEMONSTRATION  OBJECTIVES 


Purpose: 


There  are  four  objectives  of  the  demonstration.  The  overall  purpose  of  each  is  to 
show  how  the  Alaskan  Theater's  surveillance  and  command  and  control 
environment  can  be  enhanced  by  the  Advanced  Tracking  System. 


Objectives: 

1 .  Reduce  demands  on  the  FYQ-93  ROCC  central  computer;  processing  time  and 
memory  space.  At  present,  central  computer  processing  time  and  memory  storage 
space  is  nearly  saturated;  while  memory  storage  in  the  processor  controller  is 
overloaded  to  the  point  it  "dumps"  data. 

2.  Radar  Enhancement:  Regain  loss  of  probability  of  detection,  sacrificed  at  the 
expense  of  clutter  reduction  solutions  to  the  radar.  With  the  ATS's  ability  to 
extract  true  targets  from  clutter,  the  radar  beam  could  be  lowered  thus  increasing 
the  probability  of  detection  of  low  level  targets. 

3.  Increase  probability  of  detection  by  reducing  human  variability  in  the 
detection/tracking  process.  The  ATS  serving  as  a  front  end  processor  at  the  radar 
will  automatically  detect,  establish,  and  pass  target  data  to  the  Region  Operations 
Control  Center  (ROCC).  The  presentation  of  only  true  targets  in  the  ROCC  will 
eliminate  operator  decisions  concerning  the  validity  of  targets.  This  will  increase 
the  probability  of  establishing  air  tracks.  Also  because  only  true  targets  are 
presented,  maintaining  track  continuity  will  improve  for  both  computer  tracking 
algorithms  and  when  operators  override  computer  tracking. 

4.  Role  of  the  ATS  in  Alaska:  By  analyzing  the  data  collected  during  the 
demonstration,  we  will  draw  inferences  concerning  the  utility  of  the  ATS  in 
Alaskan  Theater.  Areas  of  interest:  advantages  of  incorporating  the  ATS  at  each 
AN/FPS-1 17  in  Alaska  from  a  sensor  interoperability  standpoint;  added  measure 
of  safety  in  large  scale  exercises. 
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Procedhflnres 


1 .  The  following  procedures  serve  as  a  guide  to  ensure  successful  completion  of 
the  four  objectives  for  the  demonstration. 

2.  Three  radar  sites  (Cape  Romanzof,  Kotzebue,  and  Tatalina)  will  be  used  to 
collect  data  to  satisfy  objectives  one  and  two  (reduce  demands  on  the  FYQ-93 
ROCC  central  computer,  and  radar  enhancement).  These  sites  were  chosen 
because  they  possess  the  latest  clutter  reducing  software  (i.e.  ECP  175  and  the 
Multiscan  Detection  Processing  (MDP)).  At  different  times  each  day,  data  will  be 
collected  from  each  site  for  a  period  of  105  minutes.  Each  fifteen  minutes,  within 
the  data  collection  window,  the  radar  will  be  placed  into  a  different  configuration 
(e.g.  ECP  175  enabled  and  MDP  disabled).  Data  will  be  collected  via  data 
reduction,  later  collated  on  the  Site  Data  Sheets  (figure  1)  and  analyzed. 

3.  Data  will  be  collected  using  low  level  flights  (if  available)  in  the  ACMI  range. 
This  data  will  be  used  to  assess  the  ATS's  ability  to  enhance  the  probability  of 
detection  (Pd)  of  the 

FPS -117  radar. 

4.  A  tracking  test  will  be  used  to  meet  the  third  demonstration  objective  (increase 
probability  of  detection  by  reducing  human  variability  in  the  detection/tracking 
process).  A  operator  will  be  selected  at  random  and  given  an  area  of 
responsibility  (AOR)  to  initiate  and  maintain  track  continuity  on  all  tracks  in  the 
AOR.  The  operator  will  operate  with  and  without  the  benefits  of  the  ATS  and  all 
track  histories  will  be  recorded  via  data  reduction.  A  comparison  will  be  made  of 
the  data  collected  from  the  data  reduction  products  to  determine  whether  or  not 
an  operator  will  realize  an  increased  probability  of  detection  by  reducing  human 
variability  in  the  detection/tracking  process.  Detailed  instructions  will  be 
provided  to  each  surveillance  operator  by  the  project  coordinator— thus  insuring  a 
degree  of  procedural  standardization. 

5.  In  order  to  meet  objective  four,  the  potential  role  of  the  ATS  in  Alaska, 
inferences  will  have  to  be  drawn  based  upon  the  data  collected  in  objectives  one 
through  three. 
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TOE  ATO  IMIMDNSTOATODN  L<QNG 


The  purpose  of  the  ATS  Demonstration  Log  is  to  ensure  accurate  and  timely 
recording  of  data  during  the  demonstration. 

The  ATS  demonstration  log  shall  contain  the  following  information: 


a.  Site 

b.  Start  time 

c.  End  time 

d.  Radar  configuration: 

1 .  ECP  175  (Anomalous  Propagation)  activated  (beam  raised  or 
lowered). 

2.  MDP  "ON"  or  "OFF". 

3.  Data  count  without  the  ATS  (search  only). 

4.  Data  count  without  the  ATS  (SIF  only). 

5.  Total  data  count  without  the  ATS  (Search  and  SIF). 

6.  Data  count  with  the  ATS  (search  only). 

7.  Data  count  with  the  ATS  (SIF  only). 

8.  Total  data  count  with  the  ATS  (Search  and  SIF). 

9.  Number  of  tracks  without  the  ATS. 
10.  Number  of  tracks  with  the  ATS. 

e.  ACMI  data  for  comparison. 

f.  Record  of  ROCC  track  histories  for  comparison,  data  reduction. 
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Data  products  required  to  ensure  accurate  analysis. 

1.  ATS  demonstration  log.  This  log  will  be  maintained  at  the  Air  Surveillance 
Technician  position. 

2.  Data  reduction,  hard  copy  print  out  of  flights  within  the  ACMI.  Flights  will 
be  recorded  on  tapes  located  at  the  ACMI.  During  playback,  at  the  ACMI,  hard 
copies  of  flight  data  will  be  made. 

3.  Data  reduction  from  the  FYQ-93  (ROCC)  computer  will  be  accomplished  by 
1 1  TCW/DOP  covering  the  times  of  the  demonstration. 


4-4 
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The  System  Trouble  Log  documents  any  difficulties  experienced  during  the 
demonstration. 


Date: 


Operator 


Synopsis  of  Event: 

(If  needed,  attach  additional  sheets) 


Corrective  Action: 

(If  needed,  attach  additional  sheets) 


4-5 


78 


AITS  IDdiEQOiBSftiraitta©!) 
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The  Personnel  Familiarization  Forms  document  equipment  familiarization  received 
by  the  Litton  Data  System  representatives. 


Litton  Data  System  Representative: 

Military  Personnel  Trained 

Name  Date  Time 
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HDATTA  AUTOMATODN  OTIP3W1T  IREQUJESTT 

The  Data  Automation  Support  Request  form  enclosed  is  a  form  developed  and 
used  by  the  744  ADS  to  request  data  reduction. 
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RESOURCES 


ITEM 

FUNCTION 

SOURCE 

Advanced 

Tracker 

Litton  Data  Systems 

Tracking 

System 

(ATS) 

Tactical 

Display  for  ATS 

Litton  Data  Systems 

Display 
Console 

output 

(TDC) 

Work  space 

ROCC 

USAF 

ROCC  computer 

Data  for  ATS 

USAF 
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AIWANCM)  TTIRACIKIING  OTSTOM  IDKMKOHSTmATniON  THIMIE 

JUME 

1  OCT  91 

Time:  TBD  ATS  and  TDC  equipment  arrives  to  the  Region  Operations  Control 
Center  (ROCC). 

2-4  OCT  91 

0900-  ?  Installation  and  check  out  of  ATS  equipment. 

7-9  OCT  91 

Litton  Data  Systems  representatives  will  conduct  demonstrations  of  the  ATS  and 
providing  familiarization  to  USAF  personnel. 

10  OCT  91 

0800- ?  From  this  time  forward,  until  the  end  of  the  demonstration,  data  will  be 
collected  from  various  radar  sites  at  random  times.  This  process  will  be 
accomplished  solely  by  USAF  personnel  who  have  received  familiarization  on  7-9 
Oct  91. 

17  OCT  91 

1700  Demonstration  end:  Data  will  be  collected,  compiled,  analyzed,  with  results 
presented  in  an  after  action  report. 
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1 .  (U)  General:  This  section  provides  guidance  for  the  planning  and  conduct  of 
the  ATS  demonstration. 

2.  (U)  The  denial  of  protected  information  to  an  opposing  force  is  a  command 
responsibility.  All  commands  and  individuals  associated  with  any  aspect  of 
planning,  support,  or  execution  of  the  ATS  demonstration  plan  will  safeguard 
operational  information  in  accordance  with  the  guide  that  follows. 


ESSENTIAL  ELEMENTS  OF  FRIENDLY  INFORMATION  (U) 


SUBJECT   REQUIRING   PROTECTION 


System 
AN/FYQ  -  93 
AN/FPS-117 
ON/OFF 

AN/GYQ-51  (ATS) 


Item 

Data  Counts 

Multi  scan  Detection  Processor 

Anomalous  Propagation  ON/OFF 

Radar  Coverage 

Track  Detection  Data 


NOTE:  Taken  individually,  EEFIs  are  UNCLASSIFIED.  Taken  as  a  whole,  they 
are  classified  SECRET. 
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MSTMBUnrHGDN 


11  th  Air    Force 

CC  1 

CS  1 

CV  1 

DO  3 

XP  1 

LG  1 

ALASKAN  NORAD  REGION 

CD  1 
11th     TACTICAL  CONTROL  WING 

CC  1 

DO  10 

LG  3 

SC  2 
OTHER 

21  TFW/CC  1 

21TFW/DO  1 

Litton  Data  Systems  2 

HARRY  J.  KEELING,  Col,  USAF 

Commander,  11  Tactical  Control  Wing 


OFFICIAL 


GEORGE  A.  PAHLS,  Colonel,  USAF 
Deputy  Commander  for  Operations 
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APPENDIX  C 
LRR  ENHANCEMENTS  STAFF  STUDY 
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11th  TACTICAL  CONTROL  WING  (PACAF) 
ELMENDORFAFB    AK    99506-2270 
28  FEBRUARY  1992 


LRR  ENHANCEMENTS 
STAFF  STUDY 


PACIFIC  AIR  FORCES 

1 1TH  TACTICAL  CONTROL  WING 

ELMENDORFAFB    ALASKA     99506-2270 
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FROM:  1 1  OPG/DOXXQ 

6900  9THSTSTE301 
ELMENDORFAFB  AK  99506-2270 

SUBJECT:  LRR  ENHANCEMENTS  STAFF  STUDY 

TO: 

PROBLEM 

1.  The  11  AF  needs  to  acquire  a  system  that  will  integrate  existing  systems  i.e.,  3DS, 
AICU,  and  ACMI,  into  a  cohesive,  efficient  system,  providing  unseen  benefits  to  the 
Alaska  2000+  Range  Improvements  Project. 

Note:  A  large  proportion  of  the  attachments  for  this  staff  study  come  from  the  after 
action  reports  of  Litton  Data  Systems  and  Sensis  Corporation— copied  with  their 
permission. 

FACTORS  BEARING  ON  THE  PROBLEM 

2.  Facts. 

a.  Radar  data  from  the  AN/TPS  -  63  must  be  integrated  into  the  ROCC.  The  benefits 
from  the  additional  radar  coverage  provided  by  the  TPS-63's  needs  to  be  disseminated 
and  utilized  to  the  fullest  extent  possible. 

b.  Currently,  no  capability  exists  to  automatically  insert  flight  data  on  military  aircraft 
into  3DS.  The  NORAD  Software  Support  Facility  (NSSF)  is  working  on  CSCP  #  4569 
which  would  allow  the  automatic  tell  of  military  aircraft.  Hence,  creating  the  pathway 
needed  to  get  data  on  military  aircraft  into  3DS.  The  solution  the  NSSF  is  trying  to 
achieve  is  a  major  software  change:  no  solution  is  on  the  horizon. 

c.  Track  data  on  aircraft  not  equipped  with  ACMI  pods,  but  still  players  in  large  force 
exercises  (LFEs),  are  not  entered  into  the  ACMI. 

d.  Radar  data  on  aircraft  not  equipped  with  ACMI  pods  are  detected  by  LRRs 
surrounding  the  range;  however,  this  information  is  only  sent  to  the  ROCC.  The  idea 
here  is  to  provide  data  on  exercise  participants,  not  capable  of  being  tracked  by  ACMI, 
to  the  ACMI  via  the  LRR-ROCC/AICU-ACMI  interface  (see  atch.  1).  The  results:  a 
more  complete,  accurate  air  picture,  providing  an  added  measure  of  flight  safety  both  to 
the  military  and  civil  sectors. 

e.  The  communication  interface  between  the  LRR  and  the  ROCC  needs  to  be  made 
more  efficient.  At  present  massive  amounts  of  radar  data,  most  of  it  useless  clutter  is 
being  bulk  encrypted  and  sent  over  the  communications  circuits  to  the  ROCC  (see  atch. 
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2).   Sending  only  true  target  information,  filtered  from  the  clutter,  will  reduce  the  burden 
on  the  interface,  thereby  making  this  interface  more  efficient. 

f.  Too  much  useless  data  is  being  sent  to  the  ROCC  computer  for  processing. 
Sending  only  data  which  contains  true  target  information  will  inherently  make  the 
ROCC  computer  more  efficient.  Some  of  the  benefits  are:  free  up  available  memory, 
decrease  the  amount  of  stuff  the  computer  has  to  process,  create  greater  processor 
flexibility,  redistribute  the  processing  workload  between  PPC-A  and  PPC-B. 

g.  A  human  variability  in  the  target  detection  and  tracking  process  exists.  Increased 
reliability  of  the  radar  data,  see  mostly  true  targets  vise  mostly  false  targets,  overall  air 
surveillance  function  of  the  ROCC  more  reliable  and  efficient  (atch  3). 

h.  Systems,  i.e.,  AICU,  ACMI,  CTIS/3DS,  need  to  be  integrated  more  efficiently. 
Information  available  to  one  system  can  be  made  available  to  another  system.  For 
example:  track  data  the  AICU  receives  from  an  LRR  can  be  made  available  to  both  3DS 
and  ACMI. 

3.  ASSUMPTIONS. 

a.  Industry  has  the  capability  to  solve  the  integration  requirements  we  have 
established  during  this  study. 

b.  The  AICU  will  be  able  to  receive  track  messages  from  the  RDMS  and  route  these 
messages  back  and  forth  between  the  3DS  and  ACMI. 


4.  CRITERIA. 

a.  The  solution  must  be  able  to  reduce  the  burden  on  the  LRR-ROCC  interface. 

b.  The  data  that's  being  output  is  in  a  format  that  can  be  used  by  the  ROCC 
computer  as  well  as  the  AICU. 

c.  The  solution  must  be  able  to  integrate  and  interface  the  AN/TPS-63  radar  data  in 
the  correct  format  into  the  ROCC  where  it  will  be  routed  via  the  AICU  to  the  3DS  and 
ACMI. 

e.  The  system  must  have  an  open  architecture  design,  being  able  to  be  incorporated 
into  future  systems  (i.e.,  ROMS/RAMMS). 

f.  Versatile  user  friendly  workstations,  providing  self  diagnostics  and  real  time  radar 
status  monitoring,  as  well  as  GCI  control. 
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DISCUSSION. 

5.  The  discussion  will  take  a  look  at  how  the  Alaska  2000+  Range  Improvements 
Project  can  made  more  efficient  and  solve  some  problems  that  has  been  plaguing  Alaska 
for  years.  A  solution  will  be  offered,  tested  against  criteria,  and  shown  how  this  solution 
will  solve  the  problem. 

a.  The  last  several  years  have  seen  the  proliferation  of  stand  alone  systems.  A 
common  thread  runs  through  the  purpose  of  acquiring  these  systems:  feed  the  large 
screen  display,  essential  for  making  timely  and  accurate  command  and  control  decisions. 
With  the  shift  in  emphasis  in  missions  in  Alaska,  from  war  time  preparedness,  needed  in 
the  cold  war,  to  providing  a  premier  training  center  for  the  entire  military,  these  stand 
alone  systems  must  be  flexible  enough  to  meet  our  needs.  The  purpose  of  this  study  was 
to  see  if  there  is  a  way  of  integrating  these  stand  alone  systems  into  a  cohesive  network, 
not  only  for  the  benefit  of  the  large  screen  display  for  the  battle  commanders,  but  for  all. 

b.  The  1 1  ACW  calls  the  solution  a  Radar  Data  Management  System  (RDMS)  (atch 
1).  The  RDMS  is  installed  at  the  radar  site,  and  its  function  is  to  manage  the  data  coming 
out  of  the  radar  sites.  A  closer  examination  of  the  RDMS  and  how  it  stacks  up  to  our 
criteria  will  show  the  value  the  RDMS  will  have  in  the  Alaska  2000+  Range 
Improvements  Project. 

c.  Reduce  the  burden  on  the  LRR-ROCC  interface  is  the  first  criteria  the  RDMS  must 
be  measured  against.  The  burden  on  the  LRR-ROCC  interface  is  the  quantity  of  data 
having  to  be  transmitted  over  communications  circuits.  Most  of  the  data  transmitted  is 
useless— meaning  clutter.  The  RDMS  at  the  radar  will  contain  software  that  will 
intelligently  discriminate  between  true  targets  and  false  targets  or  clutter. 
Demonstrations  by  Litton  Data  Systems  and  Sensis  Corporation  equally  illustrated  the 
dramatic  affect  of  having  a  RDMS  system  at  the  radar  site  (see  atch  4  &  5). 

d.  Sending  quality  data  to  the  ROCC  from  the  LRR  is  a  great  first  step.  The  next 
step  is  to  ensure  the  data  that's  being  output  is  in  a  format  that  can  be  used  by  the 
ROCC  computer  as  well  as  the  AICU.  The  ROCC  computer  accepts  only  plot 
messages;whereas,  the  AICU  accepts  only  track  messages.  Again,  experience  from 
contractor  demonstrations  has  shown  there  are  systems  available  that  output  data  in 
more  than  one  format.  Such  is  the  case  with  Litton's  Advanced  Tracking  System,  and 
Sensis  Corporation's  Multi-scan  Correlator  (MSC  2000)  (atch  6  &  7).  In  atch  1,  notice 
the  RDMS  will  output  plot  messages  to  the  Q-93  computer,  while  at  the  same  time 
outputting  track  messages  to  the  AICU. 

e.  Sending  track  messages  to  the  AICU  allows  several  things  to  happen.  First,  all 
aircraft  detected  by  the  radar,  sent  via  track  message  to  the  AICU,  can  then  be  routed 
directly  to  3DS.  The  interface  will  solve  a  long  standing  problem  of  how  to  get  aircraft, 
especially  military  aircraft,  automatically  into  3DS  for  display.  Second,  track  messages 
can  also  be  routed  by  the  AICU  to  the  ACMI.  The  advantages  this  provides  is  all 
aircraft  on  the  ACMI  ranges  not  equipped  with  pods  can  still  be  displayed  on  the 
ACMI.  For  example,  B-l's  from  a  CONUS  base  fly  a  strike  mission  on  the  range  during 
Cope  Thunder.     In  the  past,  these  aircraft  would  not  even  be  seen  on  the  ACMI; 
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although,  they  are  players.  The  integration  of  the  RDMS  track  message  to  the  AICU  to 
the  ACMI  will  provide  a  more  accurate  and  realistic  picture.  By  virtue  of  being  able  to 
"see"  these  types  of  players,  adds  an  extra  measure  of  safety  for  both  the  military  and 
civilian  aviators.  Finally,  with  the  same  track  messages  going  to  3DS  and  the  ACMI,  an 
automatically  updated  picture  can  be  seen  by  viewers  of  the  ACMI  and  the  3DS. 
Recording  and  playback  capability  is  the  essential  point  here.  Debriefings  can  occur 
simultaneously  at  geographically  separated  units.  Litton  and  Sensis  both  output  radai 
plot  and  track  messages  making  integration  possible  today  (atch  8). 

f.  Integrate  the  AN/TPS-63  radar  into  the  ROCC  is  another  important  criteria.  By 
having  the  TPS-63  radars  on  the  Yukon  ranges,  will  provide  low  level  radar  coverage 
never  before  seen.  However,  if  the  data  from  this  radar  doesn't  go  anywhere,  the 
benefits  will  never  fully  be  realized.  Putting  a  RDMS  in  the  TPS-63  will  enable  radai 
plot  messages  to  be  sent  to  the  ROCC  computer  for  display  and  radar  track  messages  to 
be  received  by  the  AICU  and  routed  to  the  ACMI  and  3DS.  Increased  low  level 
coverage,  where  none  previously  existed,  increased  flight  safety,  increased  GCI 
awareness,  large  screen  and  ACMI  display  of  low  level  non  podded  targets  are  all 
advantages  of  having  an  RDMS  at  the  TPS-63.  Both  Litton  Data  Systems  and  Sensis 
Corporation  indicate  their  systems  can  be  used  to  integrate  the  TPS-63  into  the  ROCC 
(atch  9  &  10). 

g.  The  RDMS  must  be  a  system  with  an  open  architecture  design,  being  able  to 
incorporate  future  systems  (atch  11).  The  first  "new"  system  is  already  becoming  a 
reality  with  the  installation  this  year  of  the  Remote  Operator  Maintenance  System 
(ROMS). 

h.  Versatile  user  friendly  workstations  are  a  necessity.  The  RDMS  must  have  a 
workstation  (located  at  the  ROCC)  with  the  versatility  of  being  used  to  monitor  the 
status  of  the  radar  on  a  real  time  basis.  On  the  other  hand,  these  workstations  must  have 
the  flexibility  to  be  used  for  either  GCI  control,  or  as  a  recording  and  playback  station. 
The  RDMS  must  have  recording  and  playback  features  essential  to  training.  Again,  both 
Litton  and  Sensis  have  demonstrated  workstations  that  could  be  used  (atch  12  &13). 

CONCLUSION. 

6.  An  RDMS  meeting  the  all  the  criteria  providing  benefits  to  the  Alaska  2000+  Range 
Improvements  Project  does  not  exist,  not  yet,  anyway.  However,  demonstrations 
conducted  by  Litton  Data  Systems,  Sensis  Corp.,  and  GE  Syracuse,  have  clearly  shown 
that  industry  possesses  the  capability  to  meet  our  established  criteria.  A  RDMS  at  the 
LRR  would  integrate  radar  data  from  both  the  AN/FPS-1 17  and  ANATPS-63  radars  into 
the  ROCC,  3DS,  and  ACMI  while  at  the  same  time  increasing  the  efficiency  of  the  entire 
Alaska  2000+  Range  network. 
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ACTION  RECOMMENDED. 

7.   Procure  a  device,  such  as  a  RDMS,  located  at  the  LRR,  which  will  provide  the  radar 
data  management  capabilities  as  described  above. 


GEORGE  A.  PAHLS,  Colonel,  USAF 
Vice  Commander 


13  Atch 

1.  RDMS  Integration  Scheme 

2.  Radar  Data  Output  Comparison  Bar  Graph 

3.  Radar  Scope  Comparison  Data 

4.  Analytical  Data:  From  Litton  Data  Systems  Final  Rpt 

5.  Analytical  Data:  Sensis  Corp.  From  Final  Rpt 

6.  Litton  data,  stating  plot  and  track  message  output 

7.  Sensis  data,  stating  correct  message  formats 

8.  Sensis  data,  integration  capability  with  CTIS 

9.  Sensis  data,  integration  capability    with  TPS-63 

10.  Litton    Data  Systems  ltr,  integration  with  TPS-63 

11.  Sensis  Corp.  stating  open  architecture  design 

12.  Sensis  Corp.  statements  on  workstations  rec/play 

13.  Litton  statement  on  workstations,  record/playback 
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RADAR  DATA  MANAGEMENT  SYSTEM  (RDMS)  INTEGRATION  SCHEME 


LRR  FPS  117/TPS  63 


PLOT  DATA 


RDMS  =  RADAR  DATA  MANAGEMENT  SYSTEM 

AICU  =  ADVANCED  INTERFACE  CONTROL  UNIT 

ACMI  =  AIR  COMBAT  MANEUVERING  INSTRUMENTATION 

3DS     =  DIGITAL  DECISION  DISPLAY  SYSTEM 
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Atachment  3  Radar  Scope  Comparison  Data 
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Figure  1-la.   Accumulation  of  all  Target  Reports  from  Kotzebue 
Radar,  June  1991  -  Large  Concentration  of  Clutter  Reports  from  Ice 
Floes  and  Other  Sources  Makes  Manual  Track  Initiation  Difficult. 
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Figure  1-lb.   Accumulation  of  all  Target  Reports  from  Kotzebue  Radar 
as  Output  by  ATS,  June  1991  -  Most  Clutter  Reports  from  Ice  Floes 
and  Other  Sources  Were  Removed,  Making  Manual  Track  Initiation  Less 
Difficult. 
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Figure  l-2a.   Number  of  all  Target  Reports  Output  by  Kotzebue  Radar 
Each  Radar  Scan,  June  1991  -  The  ASO  Must  Make  Air  Target  Decision 
from  Over  100  Target  Reports 
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Figure  l-2b.  Number  of  all  Kotzebue  Radar  Target  Reports  Output  Each 
Radar  Scan  by  ATS,  June  1991  -  The  Number  of  Target  Reports  for  Which 
ASO  Must  Make  Air  Target  Decision  Has  Been  Reduced  from  109  to  13 
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Figure  l-3a.   Accumulation  of  all  Radar  Search  Target  Reports  from 
Kotzebue  Radar,  June  1991  -  Removal  of  IFF  Target  Reports  Gives  a 
Representation  of  Clutter 
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Figure  l-3b.   Accumulation  of  all  Radar  Search  Target  Reports  from 
Kotzebue  Radar  Output  by  ATS,  June  1991  -  There  Were  Few  True 
Targets  without  IFF  in  the  Surveillance  Region 
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Figure  l-4a.   Number  of  Kotzebue  Radar  Search  Reports  Output  Each 
Radar  Scan,  June  1991  -  Removal  of  IFF  Targets  Reduces  the  Number  of 
Targets  from  103  to  8 
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Figure  l-4b.   Number  of  Kotzebue  Radar  Search  Reports  Output 
Radar  Scan  by  ATS,  June  1991  -  Three  of  These  Reports  Are  Due 
Targets,  Others  Are  Due  to  Clutter 
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Figure  l-7a.   ATS  Track  of  F-15  Air  Interceptor  (Intruder),  Tatalina 
Radar  Site,  October  1991  -  The  ATS  Maintained  Track  Even  During 
Rapid  180  Degree  Heading  Changes 
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Figure  l-7b.   ATS  Track  of  F-15  Air  Interceptor  (Defender),  Tatalina 
Radar  Site,  October  1991  -  The  ATS  Maintained  Track  for  all  Except 
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3.  DEMONSTRATION  DATA  COLLECTION  AND  ANALYSIS 


3.1  False  Alarm  Rejection  Performance 

Captain  Pierce  supervised  the  connection  of  the  MSC  and  the  collection  of  data  from 
four  AN/FPS-117  sites:  Tatalina,  Tin  City,  Kotzebue,  and  Cape  Romanzoff.  Radar 
output  data  was  collected  from  each  site  for  a  period  of  approximately  20  minutes 
for  the  purpose  of  determining  the  false  alarm  rejection  capability  of  the  MSC.  The 
collected  data  was  analyzed  off-line  using  Data  Analysis  and  Display  Software 
(DADS)  previously  developed  by  Sensis.  A  summary  of  the  false  alarm  rejection 
results  is  presented  in  Tables  3.1.  through  3.3 

Table  3. 1        False  Alarm  Rejection  of  the  MSC  on  FPS- 1 17  Radars 


Primary  Radar  Reports 

Radar  Site 

MSC  Input 

MSC  Output 

False  Alarm 
Rejection  Ratio 

Kotzebue 
Tin  City 

Cape  Romanzoff 
Tatalina 

934 
2643 

474 
2252 

34 

88 

45 

179 

27.5  to  1 
30.0  to  1 

10.5  to  1 

12.6  to  1 

Table  3.2       False  Alarm  Rejection  of  the  MSC  on  a  2D  Radar 


Primary  Radar  Reports 

Radar  Site 

MSC  Input 

MSC  Output 

False  Alarm 
Rejection  Ratio 

Kenai 

.6,595 

236 

28tol 

Table  3.3       Comparison  of  the  MDP  AND  MSC 


Primary  Radar  Reports 

Radar  Site 

MDP  Output 

MSC  Output 

False  Alarm 
Rejection  Ratio 

Tin  City 

933 

25 

37.3  to  1 

The  DADS  program  was  also  configured  to  display  all  scans  of  the  recorded  data 
(approximately  100)  for  each  site.  Once  the  data  File  playback  had  been  completed, 
the  screen  dump  feature  of  the  DADS  program  was  used.  This  enabled  the  user  to 
"take  a  snapshot"  of  the  display  and  transfer  the  output  to  a  laser  printer  where  a  hard 
copy  is  printed.     The  diamond-shaped  symbols  with  a  plus  sign  in  the  center 
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Cape  Romanzoff  Data  with  MDP  Off  and  MS C  Off. 
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Tin  City  Data  with  MDP  On  and  MSC  Off. 
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AN/GYO-51  Advanced  Tracking  System  (ATS) 

A  High  Performance  Extractor/Tracker 
for  Improved  Air  Surveillance 


INTRODUCTION 

The  AN/GYQ-51  Advanced  Tracking'  System  (ATS)  is  a  post  processor  with  radar 
beacon  target  extractor  and  tracker  capabilities  to  significantly  improve  air 
surveillance  effectiveness.  The  ATS  has  many  advanced  design  features  that 
distinguish  it  from  conventional  extractor  equipments  to  provide  improved  small 
target  detection  in  clutter.  Its  embedded  adaptive  tracker  automatically 
creates  and  maintains  a  target  track  file  which  facilitates  interoperability; 
with  systems  requiring  filtered  plot  or  track  data  inputs.  These  include  fixed 
site  Operations  Centers  (SOCCs/ROCCs)  and  mobile  systems  like  the  AN/TYQ-23 
Modular  Control  Equipment  (MCE) .  ATS  has  been  rigorously  evaluated  and  tested 
by  the  USAF  and  selected  exclusively  as  the  discriminator/tracker  for  the  TAC 
AN/TPS-43E  and  AN/TPS-75  radars  used  with  the  USAF  MCE,  and  with  the  USAF 
aerostat  radar  program.  It  is  also  presently  employed  in  the  U.S.  Customs 
Service  Southwest  Tethered  Aerostat  (STAS)  Program.  Requirements  have  been 
established  for  56  additional  USAF  units,  18  USCS  units,  1  ROKAF  unit,  with 
emerging  requirements  for  28-30  more  ATSs.  As  such,  the  AN/GYQ-51,  Advanced 
Tracking  System,  has  become  the  USAF  and  USCS  standard  post  processor/advanced 
tracker. 

Recent  live  ATS  Demonstrations  in  Alaska  (with  the  General  Electric  AN/FPS-117) 
and  High  Rock,  Bahamas  (with  the  Westinghouse  TPS-63)  prove  its  adaptability  to 
interface  with  a  wide  variety  of  older  and  new-generation  radar,  systems. 

IMPROVED  DETECTION  AND  FALSE  ALARM  PERFORMANCE 

The  ATS  upgrades  air  surveillance  system  performance  to  improve  overall  mission 
effectiveness.  The  ATS  improves  radar  detection  performance  by  allowing  radar 
detection  processing  to  operate  at  maximum  sensitivity  and  applying  post 
detection  integration  in  the  tracker  to  extract  true  targets  from  the 
background  plot  noise.  The  radar  and  target  extractor  within  the  ATS  operate 
to  achieve  optimum  small  target  (low  RCS)  detection  by  accepting  all  the 
radar's  plots,  then  allowing  the  tracker  to  perform  track-be fore-detect 
processing  by  applying  multi-scan  integration.  This  process  uses  all  available 
plot  data  to  determine  the  presence  of  true  targets  from  amid  clutter  in  the 
surveillance  volume. 

Employing  this  two  stage  detection  process  allows  the  ATS  to  achieve  optimum 
detection  performance  without  sacrificing  false  alarm  performance. 
Conventional  extractors  do  not  employ  post  detection  integration  and 
consequently  compromise  small  target  detection  to  preserve  an  acceptable  false 
alarm  rate.  Most  of  that  conventional  false  alarm  reduction  is  accomplished 
through  "fixes"  to  the  radar  (e.g.  sector  blanking,  raising  the  elevation  beam 
angle,  raising  the  sensitivity  level) .  These  fixes  do  reduce  clutter  in  the 
affected  area,  but  at  the  price  of  not  detecting  all  of  the  real  aircraft.  The 
ATS  achieves  its  improved  detection  performance  without  these  radar  fixes, 
through  adaptive  target  extraction,  noisy  area  tracking,  and  automatic  target 
discrimination . 
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In  applications  of  the  ATS  with  the  FPS-117,  the  radar  continues  to  perform  all 
the  target  extraction  functions.  The  ATS  receives  the  radar's  plot  data  via 
the  message  output  channels.  The  FPS-117  does  not  produce  the  conventional 
radar  video  that  would  allow  the  ATS  to  perform  the  plot  extraction  process  (as 
with  the  TPS-63)  .  Instead  the  ATS  extractor  is  by-passed  and  the  plot  data 
from  the  radar  is  processed  by  the  tracker  in  the  very  same  way  it  processes 
internally  generated  plot  data.  However,  the  FPS-117  plot  extraction  function 
should  be  optimized  for  maximum  sensitivity  and  probability  of  detection  (PD) , 
with  none  of  the  frequent  fixes  which  reduce  false  alarms,  but  also  reduce  the 
opportunity  to  detect  real  aircraft  as  well. 

Noisy  Area  Tracking 

The  ATS  advanced  tracker  uses  adaptive  noisy  area  tracking  techniques  and 
automatic  target  discrimination  to  determine  the  presence  of  true  targets  and 
reject  false  alarms.  The  objective  is  to  preserve  the  optimum  PD  achieved  in 
the  extractor,  and  eliminate  false  alarms  to  an  acceptably  low  level  at  the 
output  of  the  ATS  system.  Plot  data  from  the  extractor  function  is 
automatically  processed  by  the  high  capacity,  high  performance  adaptive 
tracking  function  within  the  ATS.  Noisy  area  track  initiation  is  performed  on 
all  input  data.  The  target  behavior  model  defines  the  envelope  of  target 
dynamics  that  meet  the  true  target  criteria  used  by  the  multi-scan  track 
initiation  process.  It  is  through  this  process  that  the  true  targets  are 
discriminated  from  the  background  false  plots. 

The  track  initiation  function  is  capable  of  rejecting  large  numbers  of  false 
plots  in  the  process  of  testing  all  possible  plot  combinations  for  true  target 
characteristics.  Typically,  many  thousands  of  trial  tracks  are  created  and 
tested  in  performing  this  operation.  Most  are  rejected  as  combinations  of 
false  plots  that  exhibited  brief  periods  of  true  target  behavior.  With  few 
exceptions,  only  true  targets  in  the  surveillance  volume  become  established  ATS 
tracks.  Once  those  tracks  are  established  for  the  true  targets,  the 
established  tracks  are  used  as  the  target  true  reference  to  automatically 
discriminate  plots  from  the  radar  extractor  (i.e.,  sort  out  the  noise  and 
clutter) .  The  resulting  output  consists  of  the  true  targets  and  very  few  false 
reports. 

Automatic  Target  Discrimination 

Based  on  the  measured  false  plot  density  within  individual  surveillance 
regions,  the  ATS  determines  the  eligibility  of  plot  for  output  to  the 
Operations  Center.  In  clear  areas  (receiver  noise  only) ,  plots  are  output 
without  discrimination,  as  the  likelihood  of  plots  being  true  targets  in  these 
clutter  free  regions  is  high.  In  those  areas  where  the  false  plot  density 
indicates  a  noisy  region,  plots  must  correlate  with  established  tracks  to  be 
output  by  the  ATS.  Beacon  plots  and  their  reinforcing  radar  returns  are  always 
output  without  discrimination.  This  plot  filter  criteria  is  therefore  an 
intelligent  selection  process  based  on  measured  noise  levels  and  established 
target  tracks.  This  differs  significantly  from  the  conventional,  totally 
censored  clutter  areas  where  target  detection  goes  to  zero.  This  capability 
has  been  demonstrated  as  crucial  in  meeting  PD  and  false  alarm  performance 
requirements  in  the  latest  generation  radar  applications.  Detection  and  false 
alarm  rate  requirements  are  such  that  post  detection  processing  is  the  only 
workable  solution  for  eliminating  the  false  plots  created  by  highly  sensitive 
radar  systems  (e.q.  FPS-117s  and  AerostatsK 
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INTEROPERABILITY 

The  ATS  has  proven  essential  in  providing  a  high  PD  and  low  false  alarm  plot 
input  to  Operations  Centers  for  a  variety  of  radars.  The  SOCCs/ROCCs  for 
example,  are  not  tolerant  of  false  alarms,  as  is  true  of  many  similar  centers. 
Their  central  processors  can  handle  no  more  than  10  to  15  false  alarm  plots  per 
scan  per  radar,  in  order  to  successfully  process  all  of  the  system's  radar 
data.  Otherwise,  communications  links  from  the  remote  radars  to  the  SOCC/ROCC 
become  overloaded,  and/or  the  Q-93  computer  system's  processing  capacity 
becomes  stressed  and  slowed,  creating  significant  operational  degradation,  or 
prompting  implementation  of  those  radar  fixes  which  "unplug"  or  "sector-out" 
entire  sections  of  the  regional  surveillance  volume.  The  ATS's  capabilities  to 
reduce  clutter  at  the  radar  site,  while  increasing  PD  over  true  targets,  has 
been  field-proven  to  alleviate  many  of  these  current  systemic  SCCC/ROCC 
problems . 


CONCLUSION 

The  ATS  is  a  proven,  high  performance  post-processor/advanced  tracker  that 
improves  surveillance  performance  to  meet  small,  low  flying  target  PD 
reguirements .   By  the  use  of  automatic  and  adaptive  extraction  and  tracking, 
the  ATS  enhances  radar  performance  for  small  target  detection  and  false  plot 
control.  The  ATS  is  particularly  effective  in  high  clutter  environments  caused 
by  weather,  birds,  sea  state,  topography  and  even  ECM.  The  ATS  also  provides  a 
local  target  track  file  at  the  radar  to  support  automatic  track  data  tell  to 
Operations  Centers,  offering  substantial  improvement  in  maintaining  continuity 
of  operations  with  a  distributed  target  data  base. 

The  ATS  is  now  accepted  as  a  fielded  necessity  for  detection  and  tracking  in 
many  early  warning/air  surveillance  environments,  achieving  here-to-fore 
unobtainable  automation  of  the  surveillance  function. 
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•  Captured  "pictures"  of  the  ASD  and  printed  hard  copies  on  a  laser  jet  printer. 

•  Successfully  decoded  the  "ROCC  Status"  message  in  real-time  and  displayed 
the  results  in  man-readable  textual  format  on  the  ASD. 

•  Successfully  generated  real-time  measures  of  radar  performance  such  as: 
message  counts  per  scan;  parity  errors  (per  channel);  bad  or  unrecognized 
messages;  beacon  reinforcement  rate;  primary  reinforcement  rate;  and  track 
quality. 

•  Successfully  recorded  radar  data  for  two  sites  simultaneously  and  played  the 
data  back  for  post-mission  analysis  in  either  Plan  Position  Indicator  (PPI)  or 
Range-Height  Indicator  (RHI)  format. 

•  Provided  numerous  ASD  features  including  zoom,  offset,  reorigin,  track 
blocks,  track  vectors,  tabular  displays,  SIF  locate,  and  continuously  updated 
range  and  bearing  calculations. 

•  Using  Sensis  developed  "Data  Analysis  and  Display  Software"  (DADS), 
successfully  displayed  and  analyzed  recorded  radar  data. 

•  Rejected  false  alarms  far  better  than  the  MDP  algorithm  implemented  within 
the  DDP  —  the  MSC  further  reduced  the  false  alarm  rate  out  of  the  MDP  on  the 
AN/FPS- 1 17  at  Tin  City  by  a  factor  of  39.6  to  1. 

The  important  operational  capabilities  which  Sensis  can  provide  in  a  singularly  cost- 
effective  way  include: 

•  Enhancing  range-safety  throughout  the  Yukon  MOA  by  optimizing  the 
performance  of  the  AN/FPS- 117s  and  the  AN/TPS-63  via  a  two  step  process: 
one,  increasing  radar  sensitivity  at  low  elevations  and  through  valleys  by 
proper  selection  of  detection  thresholds,  STC  attenuation  and  elevation  beam 
pointing  angles;  and  two,  employing  MSCs  to  effectively  control  false  alarms. 

•  Integrating  ROCC  track  information  into  the  CTIS  network  by  developing  a 
special  interface  board  (a  derivative  of  the  UNIO™  board  within  the  MSC) 
which  taps  into  the  data  stream  broadcast  from  the  Display  Controller  to  the 
HMD-22(J)  console. 

•  Providing  radar  plot  and  track  data  on  an  Ethernet  LAN  where  it  can  be  picked 
off  and  processed  by  new  "open  architecture"  data  processors  and  displays 
without  interfering  with  current  ROCC  operation.  The  key  to  this  realizing  this 
capability  is  to  put  MSCs  at  the  radar  sites  to  reduce  the  amount  of  plot  data 
(by  eliminating  false  alarms)  to  the  point  that  the  track  data  also  available'from 
the  MSCs  can  be  added  and  transmitted  to  the  ROCC  over  existing  data  links. 

By  presenting  data  and  supportive  discussion,  the  following  report  sections 
substantiate  the  assertions  made  here  relative  to  the  level  of  demonstrated 
performance  and  the  potential  for  improved  operational  capability.  Estimates  of  the 
cost  and  schedule  of  several  specific  programs  proposed  to  enhance  the  ROCCs 
operational  capability  are  also  included. 
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•  Captured  "pictures"  of  the  ASD  and  printed  hard  copies  on  a  laser  jet  printer. 

•  Successfully  decoded  the  "ROCC  Status"  message  in  real-time  and  displayed 
the  results  in  man-readable  textual  format  on  the  ASD. 

•  Successfully  generated  real-time  measures  of  radar  performance  such  as: 
message  counts  per  scan;  parity  errors  (per  channel);  bad  or  unrecognized 
messages;  beacon  reinforcement  rate;  primary  reinforcement  rate;  and  track 
quality. 

•  Successfully  recorded  radar  data  for  two  sites  simultaneously  and  played  the 
data  back  for  post-mission  analysis  in  either  Plan  Position  Indicator  (PPI)  or 

Range-Height  Indicator  (RHI)  format. 

i 

•  Provided  numerous  ASD  features  including  zoom,  offset,  reorigin,  track 
blocks,  track  vectors,  tabular  displays,  SEF  locate,  and  continuously  updated 
range  and  bearing  calculations. 

•  Using  Sensis  developed  "Data  Analysis  and  Display  Software"  (DADS), 
successfully  displayed  and  analyzed  recorded  radar  data. 

•  Rejected  false  alarms  far  better  than  the  MDP  algorithm  implemented  within 
the  DDP  —  the  MSC  further  reduced  the  false  alarm  rate  out  of  the  MDP  on  the 
AN/FPS-1 17  at  Tin  City  by  a  factor  of  39.6  to  1. 

The  important  operational  capabilities  which  Sensis  can  provide  in  a  singularly  cost- 
effective  way  include: 

•  Enhancing  range-safety  throughout  the  Yukon  MOA  by  optimizing  the 
performance  of  the  AN/FPS-1 17s  and  the  AN/TPS-63  via  a  two  step  process: 
one,  increasing  radar  sensitivity  at  low  elevations  and  through  valleys  by 
proper  selection  of  detection  thresholds,  STC  attenuation  and  elevation  beam 
pointing  angles;  and  two,  employing  MSCs  to  effectively  control  false  alarms. 

•  Integrating  ROCC  track  information  into  the  CI  IS  network  by  developing  a 
special  interface  board  (a  derivative  of  the  UNIO™  board  within  the  MSC) 
which  taps  into  the  data  stream  broadcast  from  the  Display  Controller  to  the 
HMD-22(J)  console. 

•  Providing  radar  plot  and  track  data  on  an  Ethernet  LAN  where  it  can  be  picked 
off  and  processed  by  new  "open  architecture"  data  processors  and  displays 
without  interfering  with  current  ROCC  operation.  The  key  to  this  realizing  this 
capability  is  to  put  MSCs  at  the  radar  sites  to  reduce  the  amount  of  plot  data 
(by  eliminating  false  alarms)  to  the  point  that  the  track  data  also  availablc'from 
the  MSCs  can  be  added  and  transmitted  to  the  ROCC  over  existing  data  links. 

By  presenting  data  and  supportive  discussion,  the  following  report  sections 
substantiate  the  assertions  made  here  relative  to  die  level  of  demonstrated 
performance  and  the  potential  for  improved  operational  capability.  Estimates  of  the 
cost  and  schedule  of  several  specific  programs  proposed  to  enhance  the  ROCCs 
operational  capability  are  also  included. 
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4.2  AN/TPS-63  Improvement 

The  key  to  improving  the  performance  of  the  AN-TPS-63  is  simply  to  reduce  the 
number  of  false  alarms  it  generates  through  the  use  of  the  MSC.  Our  conservative 
estimate  is  that  the  MSC  will  reduce  the  AN/TPS-63's  false  alarm  rate  by  more  than 
10  to  1.  This  judgment  reflects  the  performance  which  the  MSC  achieved  on  the 
ARSR-3  at  Kenai  (i.e.,  a  28  to  1  false  alarm  rejection  ratio)  and  our  experience  with 
an  aerostat-borne  AN/TPS-63  in  Yuma  Arizona.  The  MSC  cut  the  false  alarm  rate 
out  of  the  Yuma  radar  by  15  to  1,  and  because  the  moving  aerostat  has  line-of-sight 
to  a  great  deal  of  vehicular  traffic,  the  problem  is  far  more  difficult.  An  example  of 
the  performance  witnessed  on  the  Yuma  AN/TPS-63  is  provided  in  Figures  4.8  and 
4.9.  They  indicate  the  multi-scan  search-only  outputs  at  the  input  and  output  of  the 
MSC,  respectively. 

Two  unique  interfaces  are  available  to  handle  AN/TPS-63  plot  outputs.  The  USMC 
radar  utilizes  an  NTDS  parallel  interface  and  other  versions  utilize  a  three  channel 
serial  output  format.  Sensis  has  experience  with  both  interfaces.  The  MSC  has  the 
demonstrated  capability  to  accept  NTDS  format  messages,  perform  false  alarm 
rejection  processing  and  output  the  data  over  two  serial,  synchronous  data  channels 
in  the  ROCC  standard  message  format.  It  also  has  the  demonstrated  capability  to 
accept  data  from  the  aerostat  borne  radar  over  three  serial  channels,  perform  false 
alarm  rejection  processing  and  output  the  data  (with  no  data  loss)  over  two  serial, 
synchronous  data  channels  in  the  ROCC  standard  format. 
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Litton 

Data  Systems 

10  February  1992 


Captain  Mark  Pierce 
11th  Tactical  Control  Wing/DOX 
6900  9th  Street,  Suite  #301 
EJmendorf  AFB,  AK  99506 

Dear  Captain  Pierce: 

Enclosed  find  the  requested  Advanced  Tracking  System  (ATS)  specification 
information.  It  contains  detailed  ATS  specification  data  on  applicable 
government  documents/Mil -Standards,  system  and  functional  characteristics, 
logistics  and  reliability,  and  basic  Quality  Assurance/Verification  Test 
provisions . 

The  ATS  is  a  military  nomenclatured  system,  the  AN/GYQ-51,  which  has  been  in 
successful  field  use  by  the  Air  Force,  U.S.  Customs  Service  and  Air  National 
Guard  for  several  years.  In  addition,  new  ATS  systems  have  recently  been 
delivered  to  Raytheon  Corporation  for  use  in  the  Warner  Robins  AFB  PAVE  PAWS 
program,  to  the  Korean  Navy  Tactical  Data  System  (KNTDS)  and  to  General 
Electric  for  the  HYPAR  program  in  Turkey.   Fourteen  more  systems  are  in 
production  at  this  time  for  a  variety  of  other  customers,  including  NORAD. 

The  continuing  success  and  reliability  of  ATS  stems  from  Litton's  extensive 
history  of  providing  state-of-the-art,  automatic  target  detection  and  tracking 
equipments  to  interface  with  various  search  radars  and  IFF  sets.  Between 
1979-85,  Radar  Beacon  Digitizers  (RBDs)  were  delivered  for  diverse  military  and 
civil  applications  in  Iceland,  Ft.  Huachuca,  several  Navy  FACFACs,  Berlin's 
Tempelhof  airport,  and  the  USMC's  Marine  Air  Traffic  Control  and  Landing  System 
(MATCAIS) .   Litton's  newest  tracker,  the  ATS,  began  development  in  1981,  and 
led  to  the  MCE  Interface  Group  (MIG)  which  performs  stand-alone  target 
extraction  and  advanced  tracking  for  the  Air  Force's  MCE  system.  The  latest 
ATS  version,  commonly  referred  to  as  "STAS"  (for  the  Southwest  Tethered 
Aerostat  program) ,  is  the  most  highly  refined  automatic  tracker  available. 

It  is  important  to  note  that,  for  potential  Alaskan  applications,  the  ATS 
interfaces  with  a  variety  of  radars  other  than  the  FPS-117.  That  includes  full 
interoperability  with  the  TPS-63  or  ARSR-3  series  radars,  not  to  mention 
tactical  radars  that  may  deploy  to  the  theater. 

Please  contact  me  with  queries,  or  any  requests  for  follow-on  data. 


Jack  Griff 
Program  Manager 
Air  Operations 
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2.  DEMONSTRATION  CONFIGURATION 

A  block  diagram  of  the  MSC  hardware  configuration  employed  in  the  demonstration 
is  shown  in  Figure  2.1.  As  shown  therein,  two  MSCs  were  used  simultaneously. 
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Figure  2.1 


Demonstration  Hardware  Configuration 


Both  units  were  placed  on  top  of  the  Digital  Switch  located  in  the  ROCC  computer 
room.  The  MSCs  were  connected  to  serial  maintenance  ports  of  the  digital  switch 
through  standard  DB25  cables.  Data  processed  and  filtered  by  the  MSCs  was 
transmitted  onto  an  Ethernet  Local  Aicn  Network  (LAN)  using  TCP/IP  protocol. 
The  cable  was  laid  under  the  floor  to  a  SUN  SPARCstation  display  and  control 
terminal  placed  next  to  a  ROCC  Radar  Display  Unit  (RJDU). 

Within  30  minutes  after  being  unpacked,  and  with  no  hardware  adjustments  or 
modifications,  the  MSC  equipment  was  properly  set  up  and  processing  live  radar 
data.  The  setup  was  accomplished  without  interrupting  or,  in  any  way,  interfering 
with  system  operation.  Jumper  wires  were  used  to  connect  the  radar  data  and  clock 
to  the  appropriate  input  and  output  pins  on  the  MSC. 


2.1  Multi-Scan  Correlator 

The  MSC  is  a  commercially  available,  in-production  processor  which  reduces  clutter 
induced  false  target  reports  without  materially  reducing  target  detectability.  In  fact, 
the  significant  reduction  in  false  alarms  realized  by  the  MSC  peimits  an  increase  in 
radar  sensitivity,  which  can  actually  increase  target  detectability.  The  hardware 
consists  of  COTS  sub-assemblies  with  the  exception  of  one  digital  board  which  was 
specifically  designed  to  efficiently  handle  a  variety  of  serial  synchronous  and 
asynchronous  message  formats.  It  utilizes  industry  standards  such  as  the  VERTX 
real-time  operating  system,  a  VME  bus  interface,  an  Ethernet  LAN,  a  RETMA 
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chassis  and  a  TAF  standard  SUN  workstation.  Display  and  processing  code  is 
written  in  "C",  a  High  Order  Language  (HOL). 

The  MSC  is  placed  between  the  radar  and  communications  modem,  and  performs  the 
correlation  processing  required  to  elinunate  reports  from  reflectors  which  don't  move 
like  true  air  targets.  Returns  which  are  eliminated  this  way  include  surface  and 
volumetric  clutter,  long-range  clutter  induced  by  anomalous  propagation  (AP),  and 
slow-moving  targets  such  as  vehicular  traffic,  birds  and  shipping  vessels,  Beacon 
and  search-reinforced  beacon  returns  are  tracked  internally  but  not  removed  from  the 
data  stream.  The  MSC  is  capable  of  handling  up  to  1600  reports  per  12  second  scan 
(twice  the  current  data  link  capacity).  A  more  detailed  description  of  the  hardware 
and  software  of  the  MSC  is  found  in  Appendix  A. 

Physically,  the  MSC  is  housed  in  a  standard  19"  RETMA  rack  mount  chassis,  5 
inches  high  and  24  inches  deep.  The  chassis  contains  an  internal  power  supply,  two 
fans,  an  MV ME  147-1  68030  processor  board  and  a  serial  interface  board.  The 
weight  is  less  than  20  pounds  and  the  power  consumption  is  less  than  150  Watts. 

2.2  Ethernet  LAN 

All  control  communications  and  data  transfer  between  the  MSCs  and  the  SUN 
workstation  is  accomplished  via  the  Ethernet  LAN.  The  modular  nature  of  the 
system  interface  provides  the  capability  to  network  several  units  together  to  obtain  as 
many  input  ports  as  necessary.  The  bandwidth  of  the  Ethernet  LAN,  1  Megabit  per 
second,  is  sufficient  to  accept  radar  inputs  from  over  100  sources  at  4800  PPS.  All 
of  the  incoming  radar  data  is  broadcast  on  the  Ethernet  LAN  where  it  is  available  to 
be  distributed  to  other  non-proprietary  computer  systems  which  may  be  used  in  the 
future. 


2.3  SUN  SPARCstation 

The  SUN  SPARCstation  is  a  high  performance  processor  with  a  high  resolution 
display  and  built-in  mass  storage  capability.  The  SUN  utilizes  the  UNIX  operating 
system  and  windowing  to  provide  an  efficient  and  convenient  method  for  defining 
and  generating  displays  and  controls. 

The  Display  and  Control  system  implementation  makes  use  of  a  keyboard  and  mouse 
or  trackball  for  operator  entry  and  selection  of  display  information.  Radar  data  and 
operator  entries  can  be  recorded  and  stored  on  the  300  Megabyte  hard  disk  and 
transferred  to  magnetic  tape  for  data  archiving. 
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2.  DEMONSTRATION  CONFIGURATION 

A  block  diagram  of  the  MSC  hardware  configuration  employed  in  the  demonstration 
is  shown  in  Figure  2.1.  As  shown  therein,  two  MSCs  were  used  simultaneously. 
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Figure  2.1 


Demonstration  Hardware  Configuration 


Both  units  were  placed  on  top  of  the  Digital  Switch  located  in  the  ROCC  computer 
room.  The  MSCs  were  connected  to  serial  maintenance  ports  of  the  digital  switch 
through  standard  DB25  cables.  Data  processed  and  filtered  by  the  MSCs  was 
transmitted  onto  an  Ethernet  Local  Area  Network  (LAN)  using  TCP/IP  protocol. 
The  cable  was  laid  under  the  floor  to  a  SUN  SPARCstation  display  and  control 
terminal  placed  next  to  a  ROCC  Radar  Display  Unit  (RDU). 

Within  30  minutes  after  being  unpacked,  and  with  no  hardware  adjustments  or 
modificadons,  the  MSC  equipment  was  properly  set  up  and  processing  live  radar 
data.  The  setup  was  accomplished  without  internipung  or,  in  any  way,  interfering 
with  system  operation.  Jumper  wires  were  used  to  connect  the  radar  data  and  clock 
to  the  appropriate  input  and  output  pins  on  the  MSC. 


2.1  Multi-Scan  Correlator 

The  MSC  is  a  commercially  available,  in-production  processor  which  reduces  clutter 
induced  false  target  reports  without  materially  reducing  target  detectability.  In  fact, 
the  significant  reduction  in  false  alarms  realized  by  die  MSC  permits  an  increase  in 
radar  sensitivity,  which  can  actually  increase  target  detectability.  The  hardware 
consists  of  COTS  sub-assemblies  with  the  exception  of  one  digital  board  which  was 
specifically  designed  to  efficiently  handle  a  variety  of  serial  synchronous  and 
asynchronous  message  formats.  It  utilizes  industry  standards  such  as  the  VERTX 
real-time  operating  system,  a  VME  bus  interface,  an  Ethernet  LAN,  a  RETMA 
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chassis  and  a  TAP  standard  SUN  workstation.  Display  and  processing  code  is 
written  in  "C",  a  High  Order  Language  (HOL). 

The  MSC  is  placed  between  the  radar  and  communications  modem,  and  performs  the 
correlation  processing  required  to  eliminate  reports  from  reflectors  which  don't  move 
like  true  air  targets.  Returns  which  arc  eliminated  this  way  include  surface  and 
volumetric  clutter,  long-range  clutter  induced  by  anomalous  propagation  (AP),  and 
slow-moving  targets  such  as  vehicular  traffic,  birds  and  shipping  vessels,  Beacon 
and  search-reinforced  beacon  returns  are  tracked  internally  but  not  removed  from  the 
data  stream.  The  MSC  is  capable  of  handling  up  to  1600  reports  per  12  second  scan 
(twice  the  current  data  link  capacity).  A  more  detailed  description  of  the  hardware 
and  software  of  the  MSC  is  found  in  Appendix  A. 

Physically,  the  MSC  is  housed  in  a  standard  19"  RETMA  rack  mount  chassis,  5 
inches  high  and 24  inches  deep.  The  chassis  contains  an  internal  power  supply,  two 
fans,  an  MVME 147-1  68030  processor  board  and  a  serial  interface  board.  The 
weight  is  less  than  20  pounds  and  the  power  consumption  is  less  than  150  Watts. 

2,2  Ethernet  LAN 

All  control  communications  and  data  transfer  between  the  MSCs  and  the  SUN 
workstation  is  accomplished  via  the  Ethernet  LAN.  The  modular  nature  of  the 
system  interface  provides  the  capability  to  network  several  units  together  to  obtain  as 
many  input  ports  as  necessary.  The  bandwidth  of  the  Ethernet  LAN,  1  Megabit  per 
second,  is  sufficient  to  accept  radar  inputs  from  over  100  sources  at  4800  PPS.  All 
of  the  incoming  radar  data  is  broadcast  on  the  Ethernet  JLAN  where  it  is  available  to 
be  distributed  to  other  non-proprietary  computer  systems  which  may  be  used  in  the 
future. 


2.3  SUN  SPARCstation 

The  SUN  SPARCstation  is  a  high  performance  processor  with  a  high  resolution 
display  and  built-in  mass  storage  capability.  The  SUN  utilizes  the  UNDC  operating 
system  and  windowing  to  provide  an  efficient  and  convenient  method  for  defining 
and  generating  displays  and  controls. 

The  Display  and  Control  system  implementation  makes  use  of  a  keyboard  and  mouse 
or  trackball  for  operator  entry  and  selection  of  display  information.  Radar  data  and 
operator  entries  can  be  recorded  and  stored  on  the  300  Megabyte  hard  disk  and 
transferred  to  magnetic  tape  for  data  archiving. 
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9,4  Recording  and  Playback  Capability 

The  Display  and  Control  station  will  permit  the  user  to  record  the  received  data  on 
the  SUN  SPARC  workstations  internal  hard  disk  memory  unit  and  playback  the  data 
under  operator  control,  in  an  operator  designated  file,  assuming  recording  is  enabled. 

The  SUN  workstation  is  equipped  with  a  300  megabyte  internal  hard  disk  drive  with 
a  150  Megabyte  back-up  magnetic  tape  drive  for  recording  and  archiving  data.  The 
Display  Station  can  be  utilized  to  record  data  from  multiple  radars  simultaneously 
.for  later  replay  and  analysis.  Additional  display  stations  can  be  added  as 
requirements  arise  and  one  of  the  Display  Stations  could  subsume  the  functions  of 
the  present  RDU  with  a  full  color  high  resolution  display. 

The  playback  of  previously  recorded  radar  messages  is  also  contulled  via  the 
Display  and  Control  Station.  The  operator  has  the  capability  to  specify  the  playback 
filename  and  to  control  the  rate  of  playback.  Hie  rate  of  playback  can  be  varied 
from  a  real-time  rate  to  approximately  10  times  the  real-time  rate.  This  feature  is 
useful  for  viewing  data  files  to  search  for  specific  characteristics  or  events,  fn 
addition,  the  file  playback  may  be  paused  and  restarted  under  operator  control. 


9.5  System  Features  Sin:  unary 

In  summary,  the  MSC  offers  the  following  features  to  the  user: 

Modular  Architecture:  supports  single-site  remote  installations  or  networking 
of  multiple  MSCs  at  the  C3T  Ccn  er 

Programmable  "Universal"  Interface  Processor:  provides  transparent 
connection  with  existing  data  link  protocols  and  the  capability  to  perform 
channel  data  conversions  through  re-bauding  and/or  channel  muxing 

Non-Volatile  Storage  of  Site-Specific  Parameters:  for  automatic  self-activation 
upon  power-up 

On-Line  Control:  permits  user  selection  of  certain  parameters  such  as  active 
and  bypass  regions,  velocity  gates  and  clutter  maps 

Embedded  Ethernet  Local  Area  Network  (LAN)  Port:  provides  a  common 
communication  path  for  growth  options  including  the  CMC-2  subsystem  and 
the  networking  of  multiple  MSCs 

Equipment  Status:  via  built-in-test  with  front-panel  indicators  and  status  report 
messages  to  the  CMC-2 

On-Line  Performance  Monitoring:  performs  real-time  performance  monitoring 
and  health  assessment  with  fault  logging  and  automatic  bypass  in  the  event  of 
a  detected  fault 


Tracker  Data  Analysis  System  (TDAS) 

The  TDAS  Is  a  PC-based  program  for  recording  and  analyzing  ATS  output  data. 
Provisions  are  made  for  recording  of  (1)  the  raw  plot  extracted  data,  (2)  the 
filtered  plot  reports  generated  by  the  ATS,  and  (3)  the  ATS  track  file  as 
output  to  users  who  require  Established  tracks  rather  than  filtered  plots. 
TDAS  is  also  used  for  analyzing  recorded  data  and  presenting  the  analyzed  data, 
for  display.   Provisions  are  made  for  hard  copy  printout  of  displayed  data. 


In  the  Alaska  ATS  Demo,  TDAS  was  used  for  generation  of  all  the  data 
presentations  for  this  report. 

A  few  of  the  TDAS  capabilities  pertinent  to  the  ATS  demonstration,  and 
subsequent  data  analysis,  are  listed  here: 

1.  Generate  a  geoposltional  display  of  radar  target  reports,  filtered 
target  reports  or  tracks.   This  display  has  most  of  the  capabilities  of 
the  normal  radar  PPI  display  including  radar  centered  display,  display 
offset  to  cursor  (ball  tab)  position,  and  range  scale  variation. 
Provisions  for  data  retention  for  any  number  of  radar  scans  is  provided. 
Provisions  are  also  made  for  stepping  through  recorded  data  one  radar 
scan  at  a  time  (with  data  retention  for  as  many  radar  scans  as  the 
operator  chooses),  clearing  of  the  display,  and  continuation  of  the 
process.   Hard  copy  printouts  can  be  made  at  any  step  in  the  process. 

2.  Generation  of  scan  sequence  diagrams  which  present  data  values  for  each 
individual  radar  scan  (up  to  250  radar  scans  on  a  single  display).   Up 
to  four  scan- sequence  diagrams  can  be  generated  at  one  time.   Values  of 
any  track  or  plot  parameter  output  by  the  ATS  can  be  displayed. 
Examples  are : 

a.  The  number  of  target  reports  output  by  the  Target  Detector  each  radar 
scan. 

b.  The  number  of  plot  reports  output  by  the  ATS  each  radar  scan. 

c.  The  number  of  tracks  output  by  the  ATS  each  radar  scan. 

d.  Speed,  heading,  height  and  track  quality  for  any  ATS  output  track. 

3 .  Generation  of  histograms  of  data  maintained  in  the  ATS  track  file  and 
output  by  the  ATS  plot  filter.   For  example,  the  distribution  of  tracks 
with  speeds  In  25  mph  speed  intervals  for  speeds  from  zero  to  500  mph. 
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APPENDIX   D 

EXPERIMENT  PLAN  AND  REPORT  FORMATS 

GENERAL:  Ideally,  an  experiment  is  reported  on  in  two  documents,  an 
experimental  plan  that  lays  the  foundation,  and  an  experimental  report  that  tells 
what  actually  took  place  and  what  the  results  were.  For  CC4003  we  do  not 
have  time  to  produce  elaborate  experimental  plans,  so  we  move  some  of  the 
information  into  the  report.  The  following  outline  is  suggested.  Parenthetical 
notes  identify  information  that  could  be  in  the  plan,  the  report,  or  both. 

1.  Introduction  (plan  -  summarize  in  report) 

A.  Introduce  the  team 

B.  Purpose 

(1)  The  real  world  problem  the  experiment  will  help  solve 

(2)  The  specific  questions  the  experiment  seeks  to  answer 

(3)  The  approach 

(4)  Anticipated  Results 

C.  Scope  of  Experiment 

2.  Experimental  Design  (plan  --summarize  in  report) 

A.  Setup 

(1)  Physical 

(2)  Test  subjects 

(3)  Special  Equipment 

(4)  Schedule  of  Trials 

B.  Hypotheses 

C.  Assumptions 

D.  Statistical  Design  of  Experiment 
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E.  Measures 

F.  Instrumentation 

G.  Testing  &  Pilot  Trials 

3.  Data  Description  (both:  plan  --  what  will  be  collected,  assumptions; 
report  —  details) 

A.  Example  of  raw  data 

B.  Data  problems 

C.  Data  coding  scheme 

D.  Data  table 

E.  Data  Reduction 

4.  Analysis  (both:  plan  —  how  it  will  be  performed;  report  -  details) 

A.  Analysis  Plan  (plan  —  summarize  in  report) 

B.  Methodology  (report) 

C.  Results  of  Analysis        (report) 

D.  Any  additional  assumptions  that  were  required  (report) 

5.  Conclusions  (report) 

A.  Hypotheses  Results  --  Interpretations 

B.  Other  Interpretations 

C.  Real  world  meaning  of  results 

D.  Experiment  Summary 

6.  Recommendations  (report) 

A.  Changes  to  the  experiment 

B.  Continuation  of  the  Experiment 
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APPENDIX   E 


EVALUATION  OF  STUDIES  PARADIGM 


Evaluating     Studies 
CC4003--January       1994 


Simple 

short,  graphic,  minimum 

OBJECTIVE 

clear  &  relevant 

easy  to  understand, 
unambiguous,  appropriate 

ASSUMPTIONS 

reasonable 

logical,  realistic,  sensible, 
minimum 

HYPOTHESES 

germane 

testing  them  accomplishes 
objective 

EXPERIMENTAL 
DESIGN 

comprehensive 
but  executable 

pertinent,  observable, 
practical 

MEASURES 

quantifiable 

and  discriminatory 

numerical,  accurate, 
significant,   important 

DATA 

replicable 

precise,  well-defined, 
reliable 

ANALYSIS 

correct  and 
unbiased 

careful,   scientific, 
statistical,    reality 

CONCLUSIONS 

supportable 

appropriate,  valid,  logical 

Consistent 

compatibility,  direct 
linkage,  pertinent 
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DEFINITIONS  OF  CHARACTERISTICS-EVALUATION  PARADIGM 


The  objective  is  clear  and  relevant  if  it  is  easy  to  understand,  unambiguous 
and  appropriate  to  the  goal  of  the  study. 

The  study  is  simple  if  it  is  short  and  illustrative,  has  a  minimum  number  of 
assumptions,  alternatives,  hypotheses  and  measures  plus  easily  understood 
analysis  and  conclusions. 

The  assumptions  are  reasonable  if  they  can  be  supported  logically, 
realistically  and  sensibly.  They  should  be  the  minimum  necessary  to  limit  the 
scope  of  the  problem  to  viable  choices. 

The  experimental  design  is  comprehensive  but  executable  if  it  tests  all  the 
pertinent  hypotheses  with  measurable  data  by  a  practical  analysis  plan. 

A  hypothesis  is  germane  if  its  test  efficiently  accomplishes  the  objective  of 
the  study. 

Measures  are  quantifiable  if  they  provide  numerical  values  that  accurately 
reflect  the  alternatives.  The  are  discriminating  if  significant  differences  in  then- 
values  reflect  important  differences  in  the  alternatives. 

Data  is  replicable  if  it  is  precisely  defined  and  accurately  measured  and 
processed.  A  test  of  replicability  is  if  repeated  trials  would  produce  the  same  data. 

An  analysis  is  correct  if  it  carefully  tests  data  with  statistically  sound  methods. 
It  is  unbiased  if  the  assumptions  and  methods  applied  reflect  reality. 

The  conclusions  are  supportable  if  they  are  valid  results  of  an  analysis  that 
follows  logically  from  the  study. 

The  study  is  consistent  if  there  is  compatibility  in  scope  and  detail  at  each 
step.  In  addition,  direct  linkage  between  the  steps  must  maintain  continuity  with 
the  objective. 
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APPENDIX  F 


REVISED  EXPERIMENTAL  PLAN 
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I.  REVISED  INTRODUCTION 

A.  PROJECT  MANAGEMENT 

The  1 1  ACW  is  going  to  conduct  an  experiment  to  assess  the  capabilities  of 
the  Litton  Data  Systems  Advanced  Tracking  System  (ATS). 

1.  In  order  to  manage  the  demonstration,  an  Advanced  Tracking  System 
Project  Management  Team  was  established.  The  demonstration  team  leader  is 
1  lm  ACW/DOX;  other  team  members  are  listed  in  paragraph  three. 

2.  The  following  agencies  are  participating  or  providing  support  for  the 
ATS  demonstration: 

a.  11ACW/DOXXQ     --  Capt  Pierce 

b.  11ACW/DOP  --  SSgtMyher 

c.  11  ACW/LGOR        --  MSgtDeLuca 

d.  1 1  ACW/LGKC        -  MSgt  Shuler 

e.  11  ACW/LGKM        --  TSgtCushman 

f.  744ADS/DOO  --  Lt  McNeil 

g.  21stTFW/DOW       --  Capt  Hill 

B.  PURPOSE 

The  purpose  of  this  experiment  will  be  to  determine  whether  or  not  Litton 
Data  Systems  Advanced  Tracking  System  (ATS)  is  capable  of  more  efficiently 
managing  the  flow  of  radar  data  coming  into  the  Region  Operations  Control 
Center  (ROCC).  Managing  the  radar  data  more  efficiently  will  enable  the 
operators  to  realize  the  full  operational  capabilities  of  the  new  radars  without 
overloading  the  ROCC  Central  Computer.  As  a  result,  the  demonstration  is 
designed  to  provide  insight  into  the  radar  data  management  problems  being 
experienced  by  the  Alaska  NORAD  Region  (ANR)  and  the  ability  of  the  ATS  to 


125 


solve  them.  Specifically,  the  ANR  would  like  to  determine  the  validity  of  the 
claims  of  the  ATS's  capabilities.  For  example:  Can  the  ATS  accurately 
discriminate  true  targets  from  clutter?  From  an  air  defense  point  of  view,  can  the 
ATS  provide  target  discrimination  without  target  elimination? 

The  demonstration  will  select  three  sites,  all  equipped  with  ECPs.  All  of 
the  ECPs  will  be  turned  off  for  the  demonstration  in  order  to  reestablish  the  radars 
to  a  fully  sensitized  state.  The  ATS  must  prove  that  it  is  capable  of  handling  the 
data  workload  from  a  fully  sensitized  radar. 

The  ATS  will  not  be  installed  at  the  selected  radars  because  of  the  logistics 
and  prohibitive  costs.  Instead,  the  ATS  equipment  will  be  installed  in  the  ROCC. 

The  ATS  will  be  connected  at  the  ROCC  Communications  Segment,  the 
point  where  the  data  arrives  in  the  ROCC  via  satellite  from  the  selected  radars. 
The  data  arriving  at  this  point  has  not  yet  been  processed  by  the  ROCC  central 
computer.  Therefore,  connecting  the  ATS  at  the  ROCC  Communications 
Segment  simulates  the  ATS  processing  the  data  at  the  radar  site. 

The  next  step  will  be  to  compare  the  amount  of  data  a  fully  sensitized  radar 
forwards  with  the  amount  of  data  a  fully  sensitized  radar  equipped  with  an  ATS 
forwards.  The  ROCC  provides  the  capability  to  simultaneously  measure  ATS  and 
non-ATS  data  from  the  same  site  (see  Figure  4)[Ref.  10].  The  reduction  in  data, 
and  the  quality  of  the  data  must  be  analyzed  simultaneously.  The  ATS  may  live 
up  to  its  clutter  reduction  claims,  but  the  data  has  to  be  analyzed  to  ensure  that 
the  ATS  is  not  eliminating  or  erroneously  manipulating  valid  air  targets  necessary 
for  air  defense  and  air  training  missions.  In  order  to  verify  the  target  data,  a 
specific  number  of  aircraft,  flying  different  profiles,  will  be  used.  The  experiment 
will  be  designed  to  control  the  number  of  aircraft.  However,  the  actual  flight 
profiles  along  with  the  number  of  aircraft  used  during  the  test  will  only  be  known 
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by  a  third  party,  disassociated  from  the  experiment.  Only  after  the  experiment  is 
completed  and  the  analysis  turned  in  will  the  aircraft  portion  of  the  experiment  be 
revealed.  The  purpose  of  this  single  blind  design  is  to  eliminate  any  bias  from  the 
experimenters. 

The  anticipated  results:  Measurable  differences  between  the  quantity  of 
data  received  from  a  radar  in  a  fully  sensitized  state,  when  equipped  with  an  ATS 
and  when  not,  are  expected.  However,  whether  or  not  the  ATS  eliminates  valid 
air  targets  remains  to  be  seen. 

C.  THE  SCOPE  OF  THE  DEMONSTRATION: 

This  demonstration  will  provide  the  information  the  11th  AF  Commander 
needs  to  decide  whether  or  not  the  ATS  (or  other  associated  Radar  Data 
Management  System)  provides  viable  solutions  to  the  radar  data  management 
problems  experienced  by  the  ANR.  There  are  several  potential  benefits  if  the  ATS 
demonstration  proves  successful.  First,  ECPs  can  be  eliminated  from  the  radars, 
restoring  them  to  their  full  operational  capability,  and  providing  greater  low  level 
detection  of  aircraft  and  missiles.  In  the  area  of  training,  the  improved  detection 
capability  will  increase  the  radar  coverage  in  the  low  level  training  areas. 

Benefits  should  also  be  seen  in  the  ROCC  central  computer.  Installing  the 
ATS  at  the  radar  head  will  decentralize  the  computer  processing.  The  ATS  will 
discriminate  between  clutter  and  target  data,  dump  the  clutter  data,  and  pass  only 
the  target  data  to  the  central  computer  for  further  processing.  The  main  benefits 
to  the  central  computer  are  reductions  in  the  amount  of  processing  required  and 
the  potential  for  a  computer  overload  that  arbitrarily  dumps  data. 

The  reduced  computer  workload  will  enable  the  installation  of  additional 
radars.  This  is  important  because  PACAF  wants  to  expand  the  air  training  ranges 
in  Alaska.    The  eruption  of  Mount  Pinatubo  and  subsequent  loss  of  a  major  air 
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training  exercise  in  the  Philippines  called  Cope  Thunder  required  more  training, 
and  thus  an  expansion  of  the  air  training  ranges,  in  Alaska.  As  a  result,  additional 
radars  are  needed  to  provide  coverage  in  these  new  training  areas.  However,  the 
additional  data  from  these  sites  could  easily  create  an  overload  condition  in  the 
computer.  Dumping  data  during  a  major  exercise  would  seriously  jeopardize  the 
safety  of  the  entire  mission. 

The  life  of  the  central  computer  will  be  prolonged  because  the  amount  of  data 
the  central  computer  has  to  process  will  be  eased.  Thus,  the  costly  acquisition  of 
a  new  generation  ROCC  computer  can  be  delayed. 
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II.     REVISED  EXPERIMENTAL  DESIGN 

A.  SETUP: 

1.  Physical 

The  physical  setup  is  the  crux  of  the  entire  experiment.  The 
demonstration  would  not  be  possible  if  not  for  the  flexibility  of  the  ROCC 
system.  Figure  4  depicts  the  actual  physical  interface  between  the  ROCC  and  the 
ATS  equipment.  The  following  is  a  discussion  of  the  ROCC- ATS  interface. 

The  digital  switch  provides  the  interconnection  between  the 
communication  channels  from  the  radar  sites  and  the  central  computer.  The  radar 
data  at  the  digital  switch  has  not  been  processed  by  the  ROCC  computer.  Each 
radar  sends  its  data  to  the  digital  switch  at  the  ROCC  via  three  2.4  kbps 
communications  channels.  These  channels  feed  directly  into  three  ports 
associated  with  each  radar  at  the  digital  switch.  Two  ports,  designated  A  and  B, 
are  normally  interconnected  with  the  primary  and  standby  central  computers. 
The  third  port,  designated  C,  is  available  for  recording  data  on  a  noninterference 
basis.  This  is  the  location  of  the  ROCC-ATS  interface.  At  the  same  time,  the  cable 
from  port  B  of  the  Digital  Switch  to  the  central  computer  will  be  disconnected  at 
the  computer,  and  the  output  from  the  ATS  will  be  connected  to  the  standby 
computer  at  its  input  channel  (Figure  4). 

2.  Factors: 

There  are  three  experimental  factors:  radar  sites  at  three  levels;  Litton 
ATS  at  two  levels;  and  analysis  equipment  at  two  levels.  Three  radar  sites  have 
been  chosen:  Kotzebue,  Cape  Romanzof,  and  Tatalina  (Figure  2).  The  ATS  radar 
processing  hardware/software  is  either  on  or  off.  Finally,  the  equipment  being 
used  to  analyze  the  data  is  either  third  party,  Air  Force  approved,  test  and 
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evaluation  equipment  or  the  Litton  Data  Systems  analysis  program  associated 
with  the  ATS. 

3.  Special  Equipment: 

a.  The  ATS  consists  of  all  of  the  interface  hardware  as  well  as  the 
software  to  conduct  the  data  collection  and  analysis.  All  will  be  provided  by 
Litton  Data  Systems. 

b.  A  neutral  third  party  will  be  involved  to  provide  balance  and 
legitimacy  to  the  experiment.  This  third  party  (i.e.  84th  Radar  Evaluation 
Squadron)  must  conduct  the  experiment  in  parallel  with  Litton.  They  must 
interface  at  the  same  location,  with  the  same  site,  and  at  the  same  time.  They  will 
collect  the  data,  analyze  it,  and  compare  their  results  with  Litton's  experimental 
results. 

c.  Aircraft  will  be  dedicated  to  fly  certain  profiles  within  the  coverage 
limits  of  the  selected  radar. 

4.  Schedule  of  Trials: 

The  schedule  of  trials  will  not  be  formally  addressed  in  this  sample  plan. 
The  driving  factor  for  the  schedule  of  trials  will  be  the  availability  of  the  aircraft 
needed  to  fly  the  flight  profiles.  Making  up  a  schedule  of  trials  does  not  provide 
sufficient  lessons  learned. 

B.    HYPOTHESES: 

1  a.  There  is  no  difference  between  the  amount  of  data  forwarded  by  the  fully 
sensitized  radar  and  the  radar  equipped  with  the  ATS. 

lb.  The  alternative:  There  is  a  difference  between  the  amount  of  data 
forwarded  by  the  fully  sensitized  radar  and  the  radar  equipped  with  the  ATS. 
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2a.  There  is  no  difference  between  the  number  of  targets  reported  by  the 
ROCC  computer  combined  with  the  fully  sensitized  radar  and  the  radar  equipped 
with  an  ATS. 

2b.  There  is  a  difference  between  the  number  of  targets  reported  by  the 
ROCC  computer  combined  with  the  fully  sensitized  radar  and  the  radar  equipped 
with  an  ATS. 

3a.  There  is  no  difference  between  the  third  party  analysis  results  and  the 
Litton  Data  Systems  analysis  results. 

3b.  There  is  a  difference  between  the  third  party  analysis  results  and  the 
Litton  Data  Systems  analysis  results. 

The  first  two  hypotheses  deal  with  the  issues  of  quantity  and  quality  of  the 
data  the  ATS  produces.  Does  the  ATS  make  a  difference  in  the  amount  of  data 
being  sent  back  to  the  ROCC  central  computer?  And,  does  the  ATS  discriminate 
targets  from  clutter  without  eliminating  valid  air  targets? 

The  third  hypothesis  is  designed  to  validate  the  Litton  demonstration  by 
determining  whether  or  not  the  data  is  consistent  with  that  from  an  established 
test  and  evaluation  system.  The  1  ltn  ACW  staff  must  be  sure  that  the  ATS  works 
as  claimed  and  that  the  analysis  tools  that  are  part  of  the  equipment  package 
portray  the  data  in  an  accurate  manner. 

C.  ASSUMPTIONS: 

1 .  Litton's  interface  hardware  is  compatible  with  the  ROCC  equipment 

2.  The  software  will  provide  the  type  of  analysis  needed  to  answer  the 
research  questions. 

3.  The  weather  patterns  and  clutter  density  at  the  three  sites  are 
representative  of  the  (AOR)  overall. 
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4.  Both  Litton  and  the  third  party  Air  Force  evaluation  equipment  is  capable 
of  counting  and  keeping  track  of  the  number  of  targets  received  from  the  radar. 

Note:  This  assumption  is  based  on  the  claims  of  Litton  that  its  software  has 
the  capability  to  compile  this  type  of  information  [Ref.  3].  Additionally,  since  the 
ROCC  computer,  a  1970s  version  computer,  has  the  capability  to  count  the 
number  of  tracks  in  the  system,  and  is  certified,  it  is  assumed  that  the  evaluation 
equipment  exists  within  the  Air  Force,  or  else  the  ROCC  computer  could  not 
have  been  certified  in  the  first  place. 

D.  STATISTICAL  DESIGN  OF  EXPERIMENT 

The  experiment  is  a  3  X  2  X  2  factorial  single  blind  experiment.  This  means 
that  one  factor  has  three  levels  and  the  other  two  factors  each  have  two  levels. 
The  first  factor  has  three  levels  because  of  the  selection  of  three  different  radar 
sites.  The  radar  sites  associated  with  these  levels  are:  Kotzebue,  Cape  Romanzof, 
and  Tatalina.  This  factor  corresponds  to  hypotheses  one  and  two  stated  above. 
The  processing  equipment  is  the  second  factor.  The  two  levels  are  whether  or  not 
the  radar  is  utilizing  the  ATS.  This  factor  also  corresponds  to  hypotheses  one  and 
two.  The  third  factor  is  analysis  equipment.  The  two  levels  for  this  factor  are  the 
third  party  operationally  approved  equipment  and  the  Litton  analysis  equipment. 
This  factor  corresponds  to  hypothesis  number  three.  The  purpose  of  designing  a 
single  blind  experiment  is  to  eliminate  experimenter  bias  and  to  serve  as  a  means 
to  evaluate  whether  or  not  the  ATS  is  eliminating  valid  air  targets. 

E.  MEASURES 

The  experiment  is  designed  to  evaluate  the  data  forwarded  by  the  radar.  The 
data  will  be  collected  and  reduced  into  two  categories.  The  first  category  deals 
with  the  quantity  of  data  and  the  second  deals  with  the  quality  of  data  (target 
elimination).  The  categories  of  data  are  listed  below. 
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Category  one:  Quantity.  The  total  amount  of  data  processed  and  sent  by 
the  radar  to  the  ROCC,  measured  in  bits  per  second. 

Category  two:  Quality.  The  total  number  of  targets  being  sent  by  the  radar 
to  the  ROCC,  measured  by  the  total  number  of  targets  being  sent  by  the  ROCC 
computer  to  the  operator. 

F.  INSTRUMENTATION 

The  instrumentation  is  the  equipment  used  by  Litton  and  the  third  party  Air 
Force  evaluation  team  to  collect  and  analyze  the  data. 

G.  TESTING  &  PILOT  TRIALS 

Testing  will  be  done  to  ensure  the  ATS  and  third  party  equipment  can  be 
connected  to  the  ROCC  digital  switch.  Additional  testing  will  be  done  to  ensure 
the  commander  that  the  experiment  will  not  interfere  with  the  daily  operations  of 
the  ROCC. 

After  the  interfaces  are  made,  pilot  trials  will  be  run  to  ensure  that  the  data 
being  collected  is  the  correct  data  to  test  the  hypotheses.  The  researchers  will  use 
the  pilot  testing  to  practice  the  experimental  procedures.  For  planning  purposes, 
time  should  be  allotted  in  the  master  schedule  for  the  pilot  testing. 
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III.     REVISED  DATA  DESCRIPTION 

Table  1  presents  the  data  coding  scheme.  The  table  depicts  the  three  factors 
(Radar  Sites,  Analysis  Equipment,  Processing  Equipment)  along  with  their 
associated  levels.  The  number  in  parenthesis  is  the  code  assigned  to  the  particular 
level.  The  coding  scheme  will  become  apparent  when  the  table  listing  the 
different  combinations  of  trials  is  presented  below. 

TABLE   1     DATA   CODING  SCHEME 


Radar   Sites  Analysis    Equipment  Processing    Equipment 

Kotzebue  ( 1 )  Third  Party  ( 1 )  Non  ATS  ( 1 ) 

CapeRomanzoff(2)  Litton  (2)  ATS  (2) 
Tatalina  (3) 


The  data  is  divided  into  two  categories. 

Data:  Total  amount  of  data  received  from  the  radar  (X  in  bits  per  second) 
Total  number  of  tracks  ( Y) 

The  data  is  divided  into  two  categories  to  aid  in  the  analysis,  but  it  will  all  be 
collected  at  once  for  each  trial.  The  computer  software  of  the  evaluation 
equipment  will  reduce  the  data  into  the  X  and  Y  categories  designated  above. 

Table  2  depicts  the  combinations  of  levels  of  the  factors.  There  are  twelve 
combinations  of  factor  levels,  found  by  multiplying  one  factor  with  three  levels 
and  two  factors  each  with  two  levels.  Each  of  the  trials  will  be  replicated  thirty 
times  in  order  to  assume  the  data  approximates  a  normal  distribution. 
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TABLE  2     TRIALS 


Trial 

# 

Radar 

Sites 

Ai 

laly.si.s 

Equip. 

Processing  Equip.  Data  (X)  Data  (Y) 

1 

1 

1 

1 

2 

1 

1 

2 

3 

1 

2 

1 

4 

1 

2 

2 

5 

2 

1 

1 

6 

2 

1 

2 

7 

2 

2 

1 

8 

2 

2 

2 

9 

3 

1 

1 

10 

3 

1 

2 

11 

3 

2 

1 

12 

3 

2 

2 

There  are  a  several  steps  in  the  data  collection  process.  The  first  step  is  to 
collect  all  of  the  data  the  radar  is  sending  to  the  ROCC.  The  total  amount  of  data 
will  be  used  by  the  analysis  to  answer  hypotheses  one  and  three.  Target  counts 
will  then  be  compiled  using  the  software  capabilities  of  the  computers.  The  target 
counts  will  be  used  to  answer  hypothesis  number  two.  Additionally,  the  target 
count  data  can  be  used  to  provide  supplementary  information  concerning 
hypotheses  numbers  three. 
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IV.     REVISED  ANALYSIS  PLAN 

Balanced  Analysis  of  Variance  (ANOVA)  will  be  used  to  analyze  the  data. 
The  results  will  determine  the  outcome  of  the  hypotheses  tests.  As  indicated  in 
the  data  section,  there  are  two  parts  to  the  plan.  The  first  section  will  use  the 
balanced  ANOVA  to  analyze  the  total  amount  of  data.  Thirty  replications  of  each 
of  the  twelve  trials  will  be  made.  This  will  be  done  to  assume  normality.  The 
second  phase  of  the  analysis  will  also  use  balanced  ANOVA,  this  time  to  analyze 
the  thirty  replications  of  target  counts. 
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